RussellSpitzer edited a comment on issue #1463:
URL: https://github.com/apache/iceberg/issues/1463#issuecomment-693500660
Ok unfortunate thing here.
If you use "table" in your read command it will ignore all other options
passed to the builder.
So
`spark.read.option("snapshot-id",
628886310322778010L).table("hadoop_prod.ice.icetest")` is equivalent to
`spark.table("hadoop_prod.ice.icetest")`
See
https://github.com/apache/spark/blob/v3.0.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L795-L798
BUT Good news!
This is fixed in master, the fix version is marked as 3.1.0
https://issues.apache.org/jira/browse/SPARK-32592
https://github.com/apache/spark/blob/56ae95053df4afa9764df3f1d88f300896ca0183/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L832-L838
For now you'll need to use the catalog free pathway to load from a snapshot,
I'll try to see if there is another approach but
`spark.read.format("iceberg").option("snapshot-id",
"6159119813997540061").load("/tmp/ice/icetest")`
Works
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
