[
https://issues.apache.org/jira/browse/HUDI-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888079#comment-17888079
]
Y Ethan Guo commented on HUDI-8275:
-----------------------------------
Here's the hive query results after the second batch in docker demo:
{code:java}
> docker exec -it hiveserver /bin/bash
root@hiveserver:/opt# beeline -u jdbc:hive2://hiveserver:10000 --hiveconf
hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf
hive.stats.autogather=false --hivevar
hudi.hadoop.bundle=/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-hadoop-mr-bundle.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-2.8.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://hiveserver:10000
Connected to: Apache Hive (version 2.3.3)
Driver: Hive JDBC (version 2.3.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.3 by Apache Hive
0: jdbc:hive2://hiveserver:10000> add jar
/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-hadoop-mr-bundle.jar;
No rows affected (0.074 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow
group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
1 row selected (2.509 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from
stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+
1 row selected (1.862 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from
stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
1 row selected (3.058 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, ts, volume, open, close from
stock_ticks_cow where symbol = 'GOOG' order by ts;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+---------+------------+-----------+
| symbol | ts | volume | open | close |
+---------+----------------------+---------+------------+-----------+
| GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+---------+----------------------+---------+------------+-----------+
2 rows selected (2.064 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, ts, volume, open, close from
stock_ticks_mor_ro where symbol = 'GOOG' order by ts;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+---------+------------+-----------+
| symbol | ts | volume | open | close |
+---------+----------------------+---------+------------+-----------+
| GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+---------+----------------------+---------+------------+-----------+
2 rows selected (2.084 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, ts, volume, open, close from
stock_ticks_mor_rt where symbol = 'GOOG' order by ts;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+---------+------------+-----------+
| symbol | ts | volume | open | close |
+---------+----------------------+---------+------------+-----------+
| GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+---------+----------------------+---------+------------+-----------+
2 rows selected (1.904 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from
stock_ticks_cow_bs group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
1 row selected (2.02 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from
stock_ticks_mor_bs_ro group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+
1 row selected (1.773 seconds)
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from
stock_ticks_mor_bs_rt group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing
statement: FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
at
org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750) (state=08S01,code=2) {code}
> Fix MOR queries on Hive in integration tests
> --------------------------------------------
>
> Key: HUDI-8275
> URL: https://issues.apache.org/jira/browse/HUDI-8275
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Y Ethan Guo
> Assignee: Jonathan Vexler
> Priority: Blocker
> Fix For: 1.0.0
>
>
> MOR queries in integration tests (ITTestHoodieDemo#testParquetDemo) on Hive
> fail to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
