amansinha100 commented on code in PR #3742:
URL: https://github.com/apache/hive/pull/3742#discussion_r1018639310
##########
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java:
##########
@@ -209,6 +214,22 @@ public FilterCompat.Filter setFilter(final JobConf conf,
MessageType schema) {
}
}
+ private MessageType getSchemaWithoutPartitionColumns(JobConf conf,
MessageType schema) {
+ String partCols = conf.get(IOConstants.PARTITION_COLUMNS);
+ if (partCols != null && partCols.length() > 0) {
+ Set<String> partitionColumns = new
HashSet<>(Arrays.asList(partCols.split(",")));
+ List<Type> newFields = new ArrayList<>();
+
+ for (Type field: schema.getFields()) {
+ if(!partitionColumns.contains(field.getName())) {
Review Comment:
Could you confirm if this comparison works in a case-insensitive way ? i.e
if the partition columns is defined as 'part_col' and the query has predicate
on 'PART_COL'.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration
conf,
LOG.info("schema.evolution.columns and schema.evolution.columns.types
not available");
}
}
+ public static void setPartitionColumnsToConf(Configuration conf,
TableScanOperator tableScanOp) {
+ TableScanDesc scanDesc = tableScanOp.getConf();
+ if (scanDesc != null && scanDesc.getTableMetadata() != null) {
+ List<String> partitionColsList =
scanDesc.getTableMetadata().getPartColNames();
+ if (!partitionColsList.isEmpty()) {
+ conf.set(IOConstants.PARTITION_COLUMNS, String.join(",",
partitionColsList));
+ }
+ } else {
+ LOG.info(IOConstants.PARTITION_COLUMNS + " not available");
Review Comment:
Is this else condition placed correctly ? Seems like the INFO message for
partition columns not available should be logged when
partitionColsList.isEmpty() returns true.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:
##########
@@ -4272,6 +4273,21 @@ public static void addTableSchemaToConf(Configuration
conf,
LOG.info("schema.evolution.columns and schema.evolution.columns.types
not available");
}
}
+ public static void setPartitionColumnsToConf(Configuration conf,
TableScanOperator tableScanOp) {
Review Comment:
nit: pls add a comment for this method. Also, the unset method is named
'InConf' whereas this one has 'ToConf'. Can both set and unset method names
use the same suffix ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
