Hi, I’ve made the changes to the user model and added an example dataset and query for the schema inference functions. Open to any further feedback and I’d really appreciate your vote if you think it looks good!
Thank you and Regards Calvin Dani On Thu, Feb 6, 2025 at 10:40 AM Mike Carey <dtab...@gmail.com> wrote: > I have put comments on the wiki - some thoughts about the user model, etc. > > On 2/4/25 7:46 AM, Calvin Dani wrote: > > Hi, > > > > The APE has been updated following the implementation of Query 3, > > current_schema(), which fetches and aggregates the most recent schema > from > > the LSM components. > > > > The updates include: > > > > Syntax of the new query > > > > A flowchart illustrating how the query works > > > > I’d love to hear your thoughts and suggestions! If you find it promising, > > I’d appreciate your vote. > > Thank you and Regards > > Calvin Dani > > > > On Wed, Dec 18, 2024 at 11:41 AM Mike Carey<dtab...@gmail.com> wrote: > > > >> Nice! > >> > >> On 12/13/24 4:32 PM, Calvin Dani wrote: > >>> Hi, > >>> > >>> Regarding the performance testing of the first query for schema > >> inference: > >>> We benchmarked it against contemporary methods, primarily Spark-based > >>> implementations, using a configuration of 2 node controllers and 8 > >>> data partitions. > >>> > >>> For a GitHub dataset of 51GB: > >>> > >>> Our approach inferred the schema in 51.6 seconds, > >>> > >>> Spark’s native implementation took 81.6 seconds, > >>> > >>> Methods by Spoth and Mior required 400+ seconds. > >>> > >>> I hope this is helpful. > >>> > >>> Regards > >>> Calvin Dani > >>> > >>> > >>> On Thu, Dec 12, 2024 at 5:27 AM Mike Carey<dtab...@gmail.com> wrote: > >>> > >>> Question - I think you were doing some perf testing - do you have > >>> perf > >>> results for these (vs. the current schema function)? > >>> > >>> On 12/5/24 12:04 PM, Calvin Dani wrote: > >>> > Hi, > >>> > > >>> > Wanted to share an update regarding the features in the APE. The > >> two > >>> > queries: > >>> > > >>> > 1. query_schema() > >>> > > >>> > 2. collection_schema() > >>> > > >>> > are now functional. The query_schema() implementation has been > >>> submitted > >>> > for review. Once that is approved, I will proceed to submit the > >>> > collection_schema() query, as it depends on the first query's > code. > >>> > > >>> > I would greatly appreciate your feedback, additional test cases, > >>> and any > >>> > thoughts you have on this APE. I’m eager to refine it further > >>> or, if it > >>> > seems like a solid starting point, to receive approval for this > >> APE. > >>> > > >>> > Thank you for your time and input! > >>> > > >>> > Regards > >>> > > >>> > Calvin Dani > >>> > > >>> > On Wed, Nov 6, 2024 at 4:06 PM Calvin > >>> Dani<calvinthomas.d...@gmail.com> > >>> > wrote: > >>> > > >>> >> Hi, > >>> >> > >>> >> The APE has been updated with those changes! > >>> >> > >>> >> Regards > >>> >> Calvin Dani > >>> >> > >>> >> On Fri, Nov 1, 2024 at 10:36 AM Mike Carey<dtab...@gmail.com> > >>> wrote: > >>> >> > >>> >>> Excellent! +1 > >>> >>> > >>> >>> On Fri, Nov 1, 2024 at 9:35 AM Calvin > >>> Dani<calvinthomas.d...@gmail.com> > >>> >>> wrote: > >>> >>> > >>> >>>> Hi, > >>> >>>> > >>> >>>> Thank you for the feedback and as per last meeting here our > >>> the changes > >>> >>>> that are incorporated to this APE. > >>> >>>> They are as follows: > >>> >>>> 1. Name of the schema inference functions > >>> >>>> 2. Schema inference functionality > >>> >>>> > >>> >>>> The summary of changes are as follows : > >>> >>>> > >>> >>>> 1. query_schema (Aggregate function that takes all > >>> records of the > >>> >>>> subquery and generates a JSON Schema), > >>> >>>> 2. collection_schema (JSON Schema translation of the > defined > >>> >>> datatypes > >>> >>>> in the metadata node) > >>> >>>> 3. current_schema (for columnar stores and converting the > >>> inferred > >>> >>>> schema for storage compaction to JSON Schema) > >>> >>>> > >>> >>>> > >>> >>>> Regards > >>> >>>> Calvin Dani > >>> >>>> > >>> >>>> > >>> >>>> On Fri, Oct 4, 2024 at 10:28 AM Mike Carey<dtab...@gmail.com > > > >>> wrote: > >>> >>>> > >>> >>>>> Great feature! I wasn't able to understand the query > >>> example(s), > >>> >>>>> though... Could those be cleaned up a little and clarified? > >>> >>>>> > >>> >>>>> Also, I think we might want two functions at the user level > >>> - one that > >>> >>>>> takes an expression as input and reports its schema, and > >>> another that > >>> >>>>> takes a dataset/collection name as input and reports its > >>> schema. The > >>> >>>>> first one would scan the results and say what the schema is; > >>> the other > >>> >>>>> would use a more efficient approach (accessing and combining > >> the > >>> >>>>> metadata from the collection's most recent LSM components in > >>> each of > >>> >>> its > >>> >>>>> partitions). > >>> >>>>> > >>> >>>>> Cheers, > >>> >>>>> > >>> >>>>> Mike > >>> >>>>> > >>> >>>>> On 10/4/24 10:13 AM, Calvin Dani wrote: > >>> >>>>>> Initiating the discussion thread proposing a new aggregate > >>> function > >>> >>> in > >>> >>>>>> AsterixDB. > >>> >>>>>> *Feature:* aggregate function to infer schema > >>> >>>>>> *Details:* This feature introduces schema inference as an > >> SQL++ > >>> >>>> function > >>> >>>>>> directly integrated into AsterixDB. It is the first > approach > >> to > >>> >>> offer > >>> >>>>>> schema inference as a native SQL++ function, allowing users > >>> to infer > >>> >>>>>> schemas for not only any dataset but also for queries and > >>> >>> subqueries. > >>> >>>> Its > >>> >>>>>> output in JSON Schema, the industry standard, produces both > >>> human > >>> >>> and > >>> >>>>>> machine-readable results, suitable for user interpretation > or > >>> >>>> integration > >>> >>>>>> into other queries or programs. > >>> >>>>>> > >>> >>>>>> Utilizing the template of array_avg() in the Built-in > >>> Function and > >>> >>>>> Function > >>> >>>>>> collection file the array_schema() was implemented. During > >> self > >>> >>>> review, a > >>> >>>>>> lot of defined aggregate functions for > >>> >>>>>> example SerializableAvgAggregateFunction > >>> >>>>>> and IntermediateAvgAggregateFunction are not being utilised > >>> during > >>> >>>>>> array_schema() query. Is it due to different use cases or > am I > >>> >>>> utilising > >>> >>>>> it > >>> >>>>>> incorrectly? > >>> >>>>>> > >>> >>>>>> Are there any resources to understand the functionality of > >>> aggregate > >>> >>>>>> functions in the implementation? > >>> >>>>>> > >>> >>>>>> *APE* > >>> >>>>>> > >>> >>> > >>> > >> > https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+8%3A+Schema+Inference+Aggregate+Functions
