liubo1022126 commented on pull request #2614: URL: https://github.com/apache/iceberg/pull/2614#issuecomment-844950381
Thanks @pvary : My problem should overlap with #2171 problem. and I will take the time to write test code for this problem In the process of I fixing problem before, I tried to replace [projectedSchema](https://github.com/apache/iceberg/blob/0.11.x/mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java#L106) with [this.inspector = IcebergObjectInspector.create(tableSchema)] **[1]**, and then I got the error `ArrayIndexOutOfBoundsException` at [getStructFieldData](https://github.com/apache/iceberg/blob/0.11.x/mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergRecordObjectInspector.java#L73-L76) too, cause by [Object o] is only part of the select columns and [StructField structField] is all columns by init, which is modification of **[1]**. I think #2171 can also solve my problem. Although I did not use tez, our fundamental problem is the same. Because I have no experience with hive, I have not found why the projectedSchema in hconf is incorrect. But what I can sure is [hconf in method param](https://github.com/apache/hive/blob/branch-2.3/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java#L419) is transfer to a Map<TableName, Conf>, two table's conf (contains hive.io.file.readcolumn.names) is same here. that's why get [selectedColumns](https://github.com/apache/iceberg/blob/0.11.x/mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java#L92) wrong. But the class variable [conf](https://github.com/apache/hive/blob/branch-2.3/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java#L184) in super is ok, contains all tables info, can refer to [here](https://github.com/apache/hive/blob/branch-2.3/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java#L350) and [here](https://github.com/apache/hive/blob/branch-2.3/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java#L354) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
