[
https://issues.apache.org/jira/browse/SPARK-41344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644061#comment-17644061
]
Pablo Langa Blanco commented on SPARK-41344:
--------------------------------------------
In this case the provider has been detected as DataSourceV2 and also implements
SupportsCatalogOptions, so if it fails at that point, it does not make sense to
try it as DataSource V1.
The CatalogV2Util.loadTable function catches NoSuchTableException,
NoSuchDatabaseException and NoSuchNamespaceException to return an option, which
makes sense in other places where it is used, but not at this point. Maybe the
best solution is to have another function that does not catch those exceptions
to use in this case and does not return an option.
> Reading V2 datasource masks underlying error
> --------------------------------------------
>
> Key: SPARK-41344
> URL: https://issues.apache.org/jira/browse/SPARK-41344
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0, 3.3.1, 3.4.0
> Reporter: Kevin Cheung
> Priority: Critical
> Attachments: image-2022-12-03-09-24-43-285.png
>
>
> In Spark 3.3,
> # DataSourceV2Utils, the loadV2Source calls:
> {*}(CatalogV2Util.loadTable(catalog, ident, timeTravel).get{*},
> Some(catalog), Some(ident)).
> # CatalogV2Util.scala, when it tries to *loadTable(x,x,x)* and it fails with
> any of these exceptions NoSuchTableException, NoSuchDatabaseException,
> NoSuchNamespaceException, it would return None
> # Coming back to DataSourceV2Utils, None was previously returned and calling
> None.get results in a cryptic error technically "correct", but the *original
> exceptions NoSuchTableException, NoSuchDatabaseException,
> NoSuchNamespaceException are thrown away.*
>
> *Ask:*
> Retain the original error and propagate this to the user. Prior to Spark 3.3,
> the *original error* was shown and this seems like a design flaw.
>
> *Sample user facing error:*
> None.get
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:529)
> at scala.None$.get(Option.scala:527)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
> at
> org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
> at scala.Option.flatMap(Option.scala:271)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
>
> *DataSourceV2Utils.scala - CatalogV2Util.loadTable(x,x,x).get*
> [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala#L137]
> *CatalogV2Util.scala - Option(catalog.asTableCatalog.loadTable(ident))*
> {*}{{*}}[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L341]
> *CatalogV2Util.scala - catching the exceptions and return None*
> [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L344]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
