pvary commented on a change in pull request #2278:
URL: https://github.com/apache/iceberg/pull/2278#discussion_r584687461
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/serde/objectinspector/IcebergTimestampWithZoneObjectInspector.java
##########
@@ -42,12 +42,13 @@ private IcebergTimestampWithZoneObjectInspector() {
@Override
public OffsetDateTime convert(Object o) {
- return o == null ? null : OffsetDateTime.ofInstant(((Timestamp)
o).toInstant(), ZoneOffset.UTC);
+ return o == null ? null : OffsetDateTime.of(((Timestamp)
o).toLocalDateTime(), ZoneOffset.UTC);
}
@Override
public Timestamp getPrimitiveJavaObject(Object o) {
- return o == null ? null : Timestamp.from(((OffsetDateTime) o).toInstant());
+ return o == null ? null :
Review comment:
With Hive3 we can not convert the Hive filter to an Iceberg filter so we
do not push it down with Hive3:
```
2021-03-01T13:49:33,168 WARN [f0a11142-f200-4819-b157-eebcff2d39ea Test
worker] hive.HiveIcebergInputFormat (HiveIcebergInputFormat.java:getSplits(64))
- Unable to create Iceberg filter, continuing without filter (will be applied
by Hive later):
java.lang.UnsupportedOperationException: CONSTANT operator is not supported
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:85)
~[iceberg-mr-1e4412e.dirty.jar:?]
at
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.generateFilterExpression(HiveIcebergFilterFactory.java:56)
~[iceberg-mr-1e4412e.dirty.jar:?]
at
org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getSplits(HiveIcebergInputFormat.java:61)
[iceberg-mr-1e4412e.dirty.jar:?]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:425)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:395)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:314)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
[hive-exec-3.1.2-core.jar:3.1.2]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
[hive-exec-3.1.2-core.jar:3.1.2]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
[hive-exec-3.1.2-core.jar:3.1.2]
at
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:460)
[hive-service-3.1.2.jar:3.1.2]
at
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:309)
[hive-service-3.1.2.jar:3.1.2]
at
org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:905)
[hive-service-3.1.2.jar:3.1.2]
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:561)
[hive-service-3.1.2.jar:3.1.2]
at
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:553)
[hive-service-3.1.2.jar:3.1.2]
at
org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:143)
[test/:?]
at
org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerTimezone.testTimestampTzQuery(TestHiveIcebergStorageHandlerTimezone.java:214)
[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.12.jar:4.12]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
[junit-4.12.jar:4.12]
at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
[junit-4.12.jar:4.12]
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[junit-4.12.jar:4.12]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
[junit-4.12.jar:4.12]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[junit-4.12.jar:4.12]
at org.junit.runners.Suite.runChild(Suite.java:128)
[junit-4.12.jar:4.12]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[junit-4.12.jar:4.12]
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
[gradle-testing-jvm-5.4.1.jar:5.4.1]
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
[gradle-testing-jvm-5.4.1.jar:5.4.1]
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
[gradle-testing-jvm-5.4.1.jar:5.4.1]
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
[gradle-testing-jvm-5.4.1.jar:5.4.1]
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
[gradle-testing-base-5.4.1.jar:5.4.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
[gradle-messaging-5.4.1.jar:5.4.1]
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) [?:?]
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
[gradle-testing-base-5.4.1.jar:5.4.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_172]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_172]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
[gradle-messaging-5.4.1.jar:5.4.1]
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
[gradle-base-services-5.4.1.jar:5.4.1]
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
[gradle-base-services-5.4.1.jar:5.4.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
[gradle-base-services-5.4.1.jar:5.4.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
```
I fully expect that we have to adjust differently for Hive3 and Hive2 when
we fix this. But I think this should be handled in a different PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
