Hi Team,
Running tests against Iceberg backed Hive tables I was trying to run this query:
SELECT * from date_test WHERE d_date='1998-02-19'
The query fails with the Exception below. Basically complains that when
filtering the Date field it expects an Integer in the Record, but finds a
LocalDate object instead.
I am confused now, because my mental model was - until now - that Iceberg
records stores data like:
LocalDate - Types.DateType,
LocalDateTime - Types.TimestampType.withoutZone,
OffsetDateTime - Types.TimestampType.withZone,
Debugging the query above I have found that the expression evaluation expects
different types in the Record defined in org.apache.iceberg.types.Type where we
define:
Date - Integer,
Timestamp - Long
Do I supposed to prepare the Record for expression evaluation, so the evaluator
could find the correct type?
What part of the puzzle I miss?
Thanks,
Peter
The exception:
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
java.lang.IllegalStateException: Not an instance of java.lang.Integer:
2130-01-20
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:499)
at
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:307)
at
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:878)
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:559)
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:551)
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:143)
... 62 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Not an
instance of java.lang.Integer: 2130-01-20
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:494)
... 67 more
Caused by: java.lang.IllegalStateException: Not an instance of
java.lang.Integer: 2130-01-20
at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:71)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:58)
at
org.apache.iceberg.expressions.BoundReference.eval(BoundReference.java:39)
at
org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:132)
at
org.apache.iceberg.expressions.Evaluator$EvalVisitor.eq(Evaluator.java:52)
at
org.apache.iceberg.expressions.ExpressionVisitors$BoundVisitor.predicate(ExpressionVisitors.java:249)
at
org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:346)
at
org.apache.iceberg.expressions.Evaluator$EvalVisitor.eval(Evaluator.java:57)
at
org.apache.iceberg.expressions.Evaluator$EvalVisitor.access$100(Evaluator.java:52)
at org.apache.iceberg.expressions.Evaluator.eval(Evaluator.java:49)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.lambda$applyResidualFiltering$0(IcebergInputFormat.java:288)
at
org.apache.iceberg.io.CloseableIterable$3.shouldKeep(CloseableIterable.java:82)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:67)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:202)
at
org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:104)
at
org.apache.iceberg.mr.mapred.MapredIcebergInputFormat$MapredIcebergRecordReader.next(MapredIcebergInputFormat.java:81)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:488)
... 71 more
