RussellSpitzer commented on issue #1756: URL: https://github.com/apache/iceberg/issues/1756#issuecomment-726072991
This occurs because of a bad error message in Spark. What it really means is that you attempted to read the table using a datasource v1 read path when the datasource in question is a v2 one. This is occuring because you are using the native spark session catalog rather than the iceberg v2 replacement session catalog. So from here you have a two options I think, You can override the session catalog with the iceberg V2 session catalog You can read the table using the v2 path explicitly, spark.read.table() On Thu, Nov 12, 2020, 5:47 AM chaiyuan2046 <[email protected]> wrote: > spark : spark-3.0.1-bin-hadoop2.7 > iceberg : iceberg-spark3-runtime-0.9.1.jar > > ============================================== > *Step one:* > bin/spark-sql > --conf spark.sql.warehouse.dir=hdfs://xxxx:8020/user/iceberg > > *Step two:* > CREATE TABLE ib_test(id bigint, data string) USING iceberg; --success > > *Step three:* > select * from ib_test; --The error blow happens： > > Caused by: org.apache.spark.sql.AnalysisException: iceberg is not a valid > Spark SQL Data Source.; > at > org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:421) > at > org.apache.spark.sql.execution.datasources.FindDataSourceTable.$anonfun$readDataSourceTable$1(DataSourceStrategy.scala:256) > at > org.sparkproject.guava.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792) > at > org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) > at > org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) > ... 96 more > > Any ideas for this question？tks > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/issues/1756>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADE2YPBNYRXGD3YDBWJ5PDSPPDM7ANCNFSM4TTFYMWQ> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
