szlta commented on a change in pull request #2586:
URL: https://github.com/apache/hive/pull/2586#discussion_r700191437
##########
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
##########
@@ -158,8 +167,11 @@ private LlapRecordReader(MapWork mapWork, JobConf job,
FileSplit split,
rbCtx = ctx != null ? ctx : LlapInputFormat.createFakeVrbCtx(mapWork);
isAcidScan = AcidUtils.isFullAcidScan(jobConf);
- TypeDescription schema = OrcInputFormat.getDesiredRowTypeDescr(
- job, isAcidScan, Integer.MAX_VALUE);
+
+ String icebergOrcSchema =
job.get(ColumnProjectionUtils.ICEBERG_ORC_SCHEMA_STRING);
Review comment:
Yeah it's unfortunate. This is the equivalent of non-LLAP but vectorized
ORC case at
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java#L80
Hive sets IOConstants.SCHEMA_EVOLUTION_COLUMNS and
SCHEMA_EVOLUTION_COLUMNS_TYPES during compile and pushes this down to
execution. These are simple string representations of Hive types so not ORC
specific, and relates to LOGICAL schema.
Iceberg - in order to support a broader schema evolution - produces the FILE
schema based on file info and logical type info, and the result is an ORC
TypeDescription instance. (See VectorizedReadUtils.handleIcebergProjection) I
have found no easy way to transform this object back into Hive types, the
conversion only exists in the other direction:
OrcInputFormat.typeDescriptionsFromHiveTypeProperty()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
