rdblue commented on a change in pull request #7: Allow custom hadoop properties
to be loaded in the Spark data source
URL: https://github.com/apache/incubator-iceberg/pull/7#discussion_r240716751
##########
File path:
spark/src/main/java/com/netflix/iceberg/spark/source/IcebergSource.java
##########
@@ -89,30 +92,51 @@ public DataSourceReader createReader(DataSourceOptions
options) {
.toUpperCase(Locale.ENGLISH));
}
- return Optional.of(new Writer(table, lazyConf(), format));
+ return Optional.of(new Writer(table, conf, format));
}
- protected Table findTable(DataSourceOptions options) {
+ protected Table findTable(DataSourceOptions options, Configuration conf) {
Optional<String> location = options.get("path");
Preconditions.checkArgument(location.isPresent(),
"Cannot open table without a location: path is not set");
- HadoopTables tables = new HadoopTables(lazyConf());
+ HadoopTables tables = new HadoopTables(conf);
return tables.load(location.get());
}
- protected SparkSession lazySparkSession() {
+ private SparkSession lazySparkSession() {
Review comment:
This is nice to have in subclasses, which is why it is protected. We use it
in `findTable` to get information about the catalog to use. Not a big deal if
it becomes private, since we can make a quick change in our add-on library and
keep track of it there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
