RussellSpitzer commented on a change in pull request #1784:
URL: https://github.com/apache/iceberg/pull/1784#discussion_r526993346
##########
File path:
spark3/src/test/java/org/apache/iceberg/actions/TestRemoveOrphanFilesAction3.java
##########
@@ -110,4 +112,54 @@ public void testSparkCatalogHiveTable() throws
TableAlreadyExistsException, NoSu
results.contains("file:" + location + "/data/trashfile"));
}
+ @Test
+ public void testSparkSessionCatalogHadoopTable() throws Exception {
+ spark.conf().set("spark.sql.catalog.spark_catalog",
"org.apache.iceberg.spark.SparkSessionCatalog");
+ spark.conf().set("spark.sql.catalog.spark_catalog.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.spark_catalog.warehouse",
tableLocation);
+ SparkSessionCatalog cat = (SparkSessionCatalog)
spark.sessionState().catalogManager().v2SessionCatalog();
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, "table");
+ Map<String, String> options = Maps.newHashMap();
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, options);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ spark.sql("INSERT INTO default.table VALUES (1,1,1)");
+
+ String location = table.table().location().replaceFirst("file:", "");
+ new File(location + "/data/trashfile").createNewFile();
+
+ List<String> results = Actions.forTable(table.table()).removeOrphanFiles()
+ .olderThan(System.currentTimeMillis() + 1000).execute();
+ Assert.assertTrue("trash file should be removed",
+ results.contains("file:" + location + "/data/trashfile"));
+ }
+
+ @Test
+ public void testSparkSessionCatalogHiveTable() throws Exception {
+ spark.conf().set("spark.sql.catalog.spark_catalog",
"org.apache.iceberg.spark.SparkSessionCatalog");
+ spark.conf().set("spark.sql.catalog.spark_catalog.type", "hive");
+ SparkSessionCatalog cat = (SparkSessionCatalog)
spark.sessionState().catalogManager().v2SessionCatalog();
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, "sessioncattest");
+ Map<String, String> options = Maps.newHashMap();
+ Transform[] transforms = {};
+ cat.dropTable(id);
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, options);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ spark.sql("INSERT INTO default.sessioncattest VALUES (1,1,1)");
+
+ String location = table.table().location().replaceFirst("file:", "");
+ new File(location + "/data/trashfile").createNewFile();
+
+ List<String> results = Actions.forTable(table.table()).removeOrphanFiles()
+ .olderThan(System.currentTimeMillis() + 1000).execute();
Review comment:
More notes for myself, Once we have direct access to to catalog manager,
let's just do
```java
String[] ns = {"default", "sessioncattest"};
Dataset.ofRows(spark,
DataSourceV2Relation.create(cat.loadTable(Identifier.of(ns, "entries")),
Option.apply(cat), Option.apply(id)));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
