peter-toth commented on PR #38640:
URL: https://github.com/apache/spark/pull/38640#issuecomment-1315293892
> @peter-toth there is an easy way to enable parquet v2: set
`spark.sql.sources.useV1SourceList` to empty.
I thought that config only controls sources when `spark.read.parquet(...)`
is used.
Seemingly DSv2 sources are not used when I try to use it through the default
catalog:
```
bin/spark-shell --conf "spark.sql.sources.useV1SourceList="
scala> sql("create table t(id int) using parquet")
res0: org.apache.spark.sql.DataFrame = []
scala> sql("select * from t").explain(true)
== Parsed Logical Plan ==
'Project [*]
+- 'UnresolvedRelation [t], [], false
== Analyzed Logical Plan ==
id: int
Project [id#21]
+- SubqueryAlias spark_catalog.default.t
+- Relation spark_catalog.default.t[id#21] parquet
== Optimized Logical Plan ==
Relation spark_catalog.default.t[id#21] parquet
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet spark_catalog.default.t[id#21] Batched: true,
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1
paths)[file:/Users/petertoth/git/apache/spark/spark-warehouse/t],
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
```
Or did I get it wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
