[ https://issues.apache.org/jira/browse/SPARK-43155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-43155: --------------------------------- Fix Version/s: (was: 3.5.0) > DataSourceV2 is hard to be implemented without following V1 > ----------------------------------------------------------- > > Key: SPARK-43155 > URL: https://issues.apache.org/jira/browse/SPARK-43155 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: PEIYUAN SUN > Priority: Major > Labels: features > Original Estimate: 672h > Remaining Estimate: 672h > > h1. Description > The current interface of DataSourceV2 becomes overly complicated than the > Spark 2.x versions. To implement under the DataSourceV2, user needs to learn > not only the V2 APIs and interfaces. But also the DataSourceV1 (as it is a > failback version). > h2. Interface Gaps > There is no easy way and clear examples on how to implement both for a new > dataSource. For example, the examples in standard spark repo like orc, > parquet, json has a FileFormat interface for V1 while all these are not > feasible to be followed since the SPI is hard-code as `DefaultSource` instead > of dynamic loading if from user provided class outside the Spark Repo. > Different data sources are not strictly following a same pattern in V1 and > not decoupled well with customized logic within it. > > h2. Loss of simple layer over different kinds of dataSource > With original V1, user can actually implement a new wrapper on top of > orc/parquet easily with Relation Interface. The DataSourceV2 again here > becomes too low level and hard to be used in this case. > > h2. No explicit guidance > The functionality interfaces are not well organized which forces the reader > spend lots of time to understand the commit history, existing patterns as > well. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org