sdf-jkl commented on PR #22908: URL: https://github.com/apache/datafusion/pull/22908#issuecomment-5445261152
@alamb @friendlymatthew > I think this is a pretty large initial PR to merge, and I suggest we do it more incrementally ... I agree that an incremental approach would simplify the review process and make it easier to move forward. > -- like a PR to add the new crate, and one of the basic json_to_variant functions for example, rather than all 8000 lines of this one Starting with that sounds good to me :+1: > How did you determine what list of functions to use? Specifically, where did variant_object_construct, et al come from? > > In general I think we should follow existing implementations, which in this case is the spark variant functions: https://downloads.apache.org/spark/docs/4.2.0-preview3/api/sql/variant-functions > > Spark doesn't seem to have this many functions.... While working on `datafusion-variant`, we considered several existing implementations, particularly `datafusion-functions-json`. This is why the proposed function set is broader than Spark's VARIANT API. It would be helpful to agree on which implementation should guide the API -- Spark, `datafusion-functions-json`, or something else. We can start with the functionality common across them and discuss naming and additional functions in subsequent PRs. > Before we merge this I think we should send a note around to the mailing list, etc to see if anyone objects to taking on the maintenance burden of this extension package. That sounds good. We can send the note once the PR containing the crate scaffolding and first JSON-parsing function is ready. @friendlymatthew let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
