paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement
metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392612998
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
##########
@@ -103,34 +110,39 @@
// whether metadata for row groups should be collected to create files,
partitions and table metadata
private final boolean collectMetadata = false;
- public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
- ParquetReaderConfig readerConfig,
- String tableName,
- Path tableLocation,
- TupleMetadata schema,
- DrillStatsTable statsTable) {
- this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
- }
+ protected BaseParquetMetadataProvider(Builder<?> builder) {
+ if (builder.entries != null) {
+ // reuse previously stored metadata
+ this.entries = builder.entries;
+ this.tableName = builder.selectionRoot != null ?
builder.selectionRoot.toUri().getPath() : "";
+ this.tableLocation = builder.selectionRoot;
+ } else if (builder.selection != null) {
+ this.entries = new ArrayList<>();
+ this.tableName = builder.selection.getSelectionRoot() != null ?
builder.selection.getSelectionRoot().toUri().getPath() : "";
+ this.tableLocation = builder.selection.getSelectionRoot();
+ } else {
+ // case of hive parquet table
+ this.entries = new ArrayList<>();
+ this.tableName = null;
+ this.tableLocation = null;
+ }
- public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
- List<ReadEntryWithPath> entries,
- String tableName,
- Path tableLocation,
- TupleMetadata schema,
- DrillStatsTable statsTable) {
- this.entries = entries == null ? new ArrayList<>() : entries;
- this.readerConfig = readerConfig == null ?
ParquetReaderConfig.getDefaultInstance() : readerConfig;
- this.tableName = tableName;
- this.tableLocation = tableLocation;
- this.schema = schema;
- this.statsTable = statsTable;
- }
+ SchemaProvider schemaProvider =
builder.metadataProviderManager.getSchemaProvider();
+ TupleMetadata schema = builder.schema;
+ // schema passed into the builder has greater priority
+ if (schema == null && schemaProvider != null) {
+ try {
+ schema = schemaProvider.read().getSchema();
Review comment:
What is the intent of the schema here? We want to plan a query.
Historically, the schema is entirely dynamic: provided at run time. With the
provided schema, the user can constrain that dynamic schema to use the provided
schema definition (which can be strict or lenient.)
With metadata, we are feeding the scan's "exhaust" (dynamically-discovered
schema) back to itself as part of the plan.
What do we expect the can do do with this schema? Ignore and continue to
produce a dynamic schema (that is, discover any new columns, omit any old
columns which have disappeared as the last of old files were deleted, etc.)
Or, do we expect the scan to adjust its new scans to use the schema from
prior scans? There is nothing in this PR to enforce such as schema. (As it
turns out, that is what I'm currently doing.)
So, if this PR has no way to use the schema, do we even care about it? Do we
need to even fetch it? Or, is the schema here needed to understand the
available metadata for partition pruning?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
