[
https://issues.apache.org/jira/browse/DRILL-7565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038274#comment-17038274
]
ASF GitHub Bot commented on DRILL-7565:
---------------------------------------
vvysotskyi commented on pull request #1985: DRILL-7565: ANALYZE TABLE ...
REFRESH METADATA does not work for empty Parquet files
URL: https://github.com/apache/drill/pull/1985#discussion_r380128453
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/metadata/MetadataAggregateHelper.java
##########
@@ -313,8 +317,19 @@ private void addColumnAggregateCalls(FieldReference
fieldRef, String fieldName)
if (interestingColumns == null || interestingColumns.contains(fieldRef))
{
// collect statistics for all or only interesting columns if they are
specified
AnalyzeColumnUtils.COLUMN_STATISTICS_FUNCTIONS.forEach((statisticsKind,
sqlKind) -> {
+ // constructs "case when is not null projectMetadataColumn then
column1 else null end" call
+ // to avoid using default values for required columns when data for
empty result is obtained
Review comment:
Thanks for pointing this. Unfortunately, we can't use a plain SQL approach
to collect metadata, since we do not have information about the schema, so we
create aggregate calls dynamically. But Drill uses inbuilt aggregate functions
for collecting summary statistics (`MIN`, `MAX`, ...).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> ANALYZE TABLE ... REFRESH METADATA does not work for empty Parquet files
> ------------------------------------------------------------------------
>
> Key: DRILL-7565
> URL: https://issues.apache.org/jira/browse/DRILL-7565
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.17.0
> Reporter: Bohdan Kazydub
> Assignee: Vova Vysotskyi
> Priority: Major
> Fix For: 1.18.0
>
>
> The following query does not create metadata for empty Parquet table:
> {code:java}
> @Test
> public void testAnalyzeEmptyParquetTable() throws Exception {
> dirTestWatcher.copyResourceToRoot(Paths.get("parquet", "empty"));
> String tableName = "parquet/empty/simple/empty_simple.parquet";
> try {
> client.alterSession(ExecConstants.METASTORE_ENABLED, true);
> testBuilder()
> .sqlQuery("ANALYZE TABLE dfs.`%s` REFRESH METADATA", tableName)
> .unOrdered()
> .baselineColumns("ok", "summary")
> .baselineValues(true, String.format("Collected / refreshed metadata
> for table [dfs.default.%s]", tableName))
> .go();
> } finally {
> run("analyze table dfs.`%s` drop metadata if exists", tableName);
> client.resetSession(ExecConstants.METASTORE_ENABLED);
> }
> }
> {code}
> but yields
> {code:java}
> java.lang.AssertionError: Different number of records returned
> Expected :1
> Actual :0
> <Click to see difference>
> at
> org.apache.drill.test.DrillTestWrapper.compareResults(DrillTestWrapper.java:862)
> at
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:567)
> at org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:171)
> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:145)
> at
> org.apache.drill.exec.store.parquet.TestEmptyParquet.testSelectWithDisabledMetastore(TestEmptyParquet.java:430)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> When changing expected result set to empty
> ({{TestBuilder#expectsEmptyResultSet()}}), {{SHOW TABLES}} command after
> {{ANALYZE TABLE ...}} does not show any table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
