aokolnychyi commented on code in PR #7228:
URL: https://github.com/apache/iceberg/pull/7228#discussion_r1157829683
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java:
##########
@@ -134,28 +135,34 @@ public Identifier[] listTables(String[] namespace) throws
NoSuchNamespaceExcepti
@Override
public Table loadTable(Identifier ident) throws NoSuchTableException {
- try {
- return icebergCatalog.loadTable(ident);
- } catch (NoSuchTableException e) {
- return getSessionCatalog().loadTable(ident);
- }
+ return loadTableInternal(ident, null, null);
}
@Override
public Table loadTable(Identifier ident, String version) throws
NoSuchTableException {
- try {
- return icebergCatalog.loadTable(ident, version);
- } catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- return getSessionCatalog().loadTable(ident, version);
- }
+ return loadTableInternal(ident, version, null);
}
@Override
public Table loadTable(Identifier ident, long timestamp) throws
NoSuchTableException {
+ return loadTableInternal(ident, null, timestamp);
+ }
+
+ private Table loadTableInternal(Identifier ident, String version, Long
timestamp)
+ throws NoSuchTableException {
try {
- return icebergCatalog.loadTable(ident, timestamp);
+ return loadTable(icebergCatalog, ident, version, timestamp);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- return getSessionCatalog().loadTable(ident, timestamp);
+ return loadTable(getSessionCatalog(), ident, version, timestamp);
+ } catch (org.apache.iceberg.exceptions.NotFoundException e) {
+ if (loadCatalogTableWhenMetadataNotFoundEnabled()) {
Review Comment:
I think we are trying to fix this problem at a wrong level. I feel Spark
logic should be as follows:
- Load Iceberg table if possible. If not possible, it is OK.
- Call icebergCatalog.drop(ident, false) to drop the pointer no matter
whether the first step was successful.
- If Iceberg table is not null and metadata file still exists, call the
cleanup action.
All we would need to do is to adapt our Iceberg catalog implementations to
drop pointers even if the metadata file does not exist. That's where we can
check if the catalog entry seems like an Iceberg table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
