HyukjinKwon commented on PR #59424: URL: https://github.com/apache/spark/pull/59424#issuecomment-6184834495
@LuciferYang Thank you, the comparison with lance-spark is very helpful. I agree with the positioning. In 51af65c63a93, the docs have a new "When to Use a Native Data Source" section, which leads with a source whose client exists only in native code, such as an in-house storage system in C++ or the C SDK of a vendor, and that Spark needs to read and write. The PR description leads with it too, and no longer lists Lance among the motivating examples. This work did not start from one specific source: the goal was to let a data source be written only in native code, which is why the examples in the docs read a range of numbers. On scope, the line I would draw is Data Source V2 itself: - Features of Data Source V2 that version 1 does not cover are in scope for later, as functions and capabilities appended to the table: aggregate pushdown, including `count(*)`, and top-N pushdown; statistics; reporting the partitioning and the sort order of the data, which storage-partitioned joins use and which is the closest to fragment-aware joins; overwrite by filter; and row-level operations with metadata columns, as `SupportsRowLevelOperations` and `SupportsDelta` do with a row ID. The Limitations section now lists top-N pushdown, row-level operations and metadata columns too. - Extensions outside of Data Source V2, such as table functions like `VECTOR_SEARCH`, SQL commands like `CREATE INDEX`, and parser and optimizer rules, are out of the scope of this interface on purpose: they plug into Spark's extension points, such as `SparkSessionExtensions`, rather than into Data Source V2. A source that needs them is better served by a JVM connector like lance-spark, which can keep sharing the native read and write path through the Arrow C stream interface, as it does now. The new section of the docs says so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
