Created a PR with the fix, but not entirely comfortable with the changes in GenericRecord.java
https://github.com/apache/iceberg/pull/2254 <https://github.com/apache/iceberg/pull/2254> If you have time and more knowledge around the expected behavior of the GenericRecord, please review. Thanks, Peter > On Feb 19, 2021, at 15:08, Peter Vary <pv...@cloudera.com> wrote: > > Hi Team, > > Running tests against Iceberg backed Hive tables I was trying to run this > query: > > SELECT * from date_test WHERE d_date='1998-02-19' > > The query fails with the Exception below. Basically complains that when > filtering the Date field it expects an Integer in the Record, but finds a > LocalDate object instead. > > I am confused now, because my mental model was - until now - that Iceberg > records stores data like: > LocalDate - Types.DateType, > LocalDateTime - Types.TimestampType.withoutZone, > OffsetDateTime - Types.TimestampType.withZone, > > Debugging the query above I have found that the expression evaluation expects > different types in the Record defined in org.apache.iceberg.types.Type where > we define: > Date - Integer, > Timestamp - Long > > Do I supposed to prepare the Record for expression evaluation, so the > evaluator could find the correct type? > What part of the puzzle I miss? > > Thanks, > Peter > > The exception: > > Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: > java.lang.IllegalStateException: Not an instance of java.lang.Integer: > 2130-01-20 > at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:499) > at > org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:307) > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:878) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:559) > at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:551) > at > org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:143) > ... 62 more > Caused by: java.io.IOException: java.lang.IllegalStateException: Not an > instance of java.lang.Integer: 2130-01-20 > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428) > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147) > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208) > at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:494) > ... 67 more > Caused by: java.lang.IllegalStateException: Not an instance of > java.lang.Integer: 2130-01-20 > at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123) > at > org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:71) > at > org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:58) > at > org.apache.iceberg.expressions.BoundReference.eval(BoundReference.java:39) > at > org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:132) > at > org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:52) > at > org.apache.iceberg.expressions.ExpressionVisitors$BoundVisitor.predicate(ExpressionVisitors.java:249) > at > org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:346) > at > org.apache.iceberg.expressions.Evaluator$EvalVisitor.eval(Evaluator.java:57) > at > org.apache.iceberg.expressions.Evaluator$EvalVisitor.access$100(Evaluator.java:52) > at org.apache.iceberg.expressions.Evaluator.eval(Evaluator.java:49) > at > org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.lambda$applyResidualFiltering$0(IcebergInputFormat.java:288) > at > org.apache.iceberg.io.CloseableIterable$3.shouldKeep(CloseableIterable.java:82) > at > org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:67) > at > org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50) > at > org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:202) > at > org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:104) > at > org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:81) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:488) > ... 71 more
