n3nash commented on a change in pull request #972: [HUDI-313] Fix select count
star error when querying a realtime table
URL: https://github.com/apache/incubator-hudi/pull/972#discussion_r339726209
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -197,10 +197,27 @@ private static synchronized Configuration
addRequiredProjectionFields(Configurat
return configuration;
}
+ /**
+ * Hive will append read columns' ids to old columns' ids during
getRecordReader. In some cases, e.g. SELECT COUNT(*),
+ * the read columns' id is an empty string and Hive will combine it with
Hoodie required projection ids and becomes
+ * e.g. ",2,0,3" and will cause an error. This method is used to avoid this
situation.
+ */
+ private static synchronized Configuration
cleanProjectionColumnIds(Configuration conf) {
Review comment:
@zhedoubushishi That makes sense. Although, the hoodie projection column ids
are added by the method addRequiredProjectionFields right below by the realtime
format (which is invoked by hive). Can we perform this check before adding
those projection columns themselves ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
