[ https://issues.apache.org/jira/browse/SPARK-41344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644087#comment-17644087 ]
Zhen Wang commented on SPARK-41344: ----------------------------------- [~planga82] Thanks for your reply, I have submitted a PR [https://github.com/apache/spark/pull/38871], can you help me review it? > Maybe the best solution is to have another function that does not catch those > exceptions to use in this case and does not return an option. Does this mean we need to add a new method in CatalogV2Util? > Reading V2 datasource masks underlying error > -------------------------------------------- > > Key: SPARK-41344 > URL: https://issues.apache.org/jira/browse/SPARK-41344 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.3.0, 3.3.1, 3.4.0 > Reporter: Kevin Cheung > Priority: Critical > Attachments: image-2022-12-03-09-24-43-285.png > > > In Spark 3.3, > # DataSourceV2Utils, the loadV2Source calls: > {*}(CatalogV2Util.loadTable(catalog, ident, timeTravel).get{*}, > Some(catalog), Some(ident)). > # CatalogV2Util.scala, when it tries to *loadTable(x,x,x)* and it fails with > any of these exceptions NoSuchTableException, NoSuchDatabaseException, > NoSuchNamespaceException, it would return None > # Coming back to DataSourceV2Utils, None was previously returned and calling > None.get results in a cryptic error technically "correct", but the *original > exceptions NoSuchTableException, NoSuchDatabaseException, > NoSuchNamespaceException are thrown away.* > > *Ask:* > Retain the original error and propagate this to the user. Prior to Spark 3.3, > the *original error* was shown and this seems like a design flaw. > > *Sample user facing error:* > None.get > java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:529) > at scala.None$.get(Option.scala:527) > at > org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129) > at > org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209) > at scala.Option.flatMap(Option.scala:271) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171) > > *DataSourceV2Utils.scala - CatalogV2Util.loadTable(x,x,x).get* > [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala#L137] > *CatalogV2Util.scala - Option(catalog.asTableCatalog.loadTable(ident))* > {*}{{*}}[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L341] > *CatalogV2Util.scala - catching the exceptions and return None* > [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L344] -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org