paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement
metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609285
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
##########
@@ -118,9 +135,59 @@ public void endBatch() {
// Fill in the null and metadata columns.
populateNonDataColumns();
}
+ if (projected) {
+ setProjectMetadata(null);
+ }
rootTuple.setRowCount(tableContainer.getRecordCount());
}
+ /**
+ * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"}
to handle current batch as
+ * a batch with metadata information only for the case when this batch is
first and empty.
+ */
Review comment:
Why? As it turns out, this kind of logic is handled in the
`ScanOperatorExec` and `ReaderState`. See
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/ReaderState.java#L444
I have to apologize, that reader code is quite tricky. It was hard to handle
all the strange things that readers can do such as hitting EOF on open (a CSV
file declared with headers but which is empty), on the first batch (a CVS file
without headers which is empty), and so on.
Can we insert this logic there somehow? Even better, does the existing
handling provide the behavior you are trying to achieve? Maybe this logic was
needed for the old scan which was less graceful at handling odd cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
