[jira] [Created] (HIVE-11369) Mapjoins in HiveServer2 fail when jmxremote is used
David Morel created HIVE-11369:
--
Summary: Mapjoins in HiveServer2 fail when jmxremote is used
Key: HIVE-11369
URL: https://issues.apache.org/jira/browse/HIVE-11369
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 1.1.0
Environment: CDH 5.4.3, Centos 6.5, java version "1.7.0_67"
Reporter: David Morel
having hive.auto.convert.join set to true works in the CLI with no issue, but
fails in HiveServer2 when jmx options are passed to the service on startup.
This (in hive-env.sh) is enough to make it fail:
{noformat}
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=8009
{noformat}
As soon as I remove the line, it works properly. I have *no*idea...
Here's the log from the service:
{noformat}
2015-07-24 17:19:27,457 INFO [HiveServer2-Handler-Pool: Thread-22]: ql.Driver
(SessionState.java:printInfo(912)) - Query ID =
hive_20150724171919_aaa88a89-dc6d-490b-821c-4eec6d4c0421
2015-07-24 17:19:27,457 INFO [HiveServer2-Handler-Pool: Thread-22]: ql.Driver
(SessionState.java:printInfo(912)) - Total jobs = 1
2015-07-24 17:19:27,465 INFO [HiveServer2-Handler-Pool: Thread-22]: ql.Driver
(Driver.java:launchTask(1638)) - Starting task [Stage-4:MAPREDLOCAL] in serial
mode
2015-07-24 17:19:27,467 INFO [HiveServer2-Handler-Pool: Thread-22]:
mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(159)) - Generating
plan file
file:/tmp/hive/8932c206-5420-4b6f-9f1f-5f1706f30df8/hive_2015-07-24_17-19-26_552_5082133674120283907-1/-local-10005/plan.xml
2015-07-24 17:19:27,625 WARN [HiveServer2-Handler-Pool: Thread-22]:
conf.HiveConf (HiveConf.java:initialize(2620)) - HiveConf of name
hive.files.umask.value does not exist
2015-07-24 17:19:27,708 INFO [HiveServer2-Handler-Pool: Thread-22]:
mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(288)) - Executing:
/usr/lib/hadoop/bin/hadoop jar /usr/lib/hive/lib/hive-common-1.1.0-cdh5.4.3.jar
org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan
file:/tmp/hive/8932c206-5420-4b6f-9f1f-5f1706f30df8/hive_2015-07-24_17-19-26_552_5082133674120283907-1/-local-10005/plan.xml
-jobconffile
file:/tmp/hive/8932c206-5420-4b6f-9f1f-5f1706f30df8/hive_2015-07-24_17-19-26_552_5082133674120283907-1/-local-10006/jobconf.xml
2015-07-24 17:19:28,499 ERROR [HiveServer2-Handler-Pool: Thread-22]: exec.Task
(SessionState.java:printError(921)) - Execution failed with exit status: 1
2015-07-24 17:19:28,500 ERROR [HiveServer2-Handler-Pool: Thread-22]: exec.Task
(SessionState.java:printError(921)) - Obtaining error information
2015-07-24 17:19:28,500 ERROR [HiveServer2-Handler-Pool: Thread-22]: exec.Task
(SessionState.java:printError(921)) -
Task failed!
Task ID:
Stage-4
Logs:
2015-07-24 17:19:28,501 ERROR [HiveServer2-Handler-Pool: Thread-22]: exec.Task
(SessionState.java:printError(921)) - /tmp/hiveserver2_manual/hive-server2.log
2015-07-24 17:19:28,501 ERROR [HiveServer2-Handler-Pool: Thread-22]:
mr.MapredLocalTask (MapredLocalTask.java:executeInChildVM(308)) - Execution
failed with exit status: 1
2015-07-24 17:19:28,518 ERROR [HiveServer2-Handler-Pool: Thread-22]: ql.Driver
(SessionState.java:printError(921)) - FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
2015-07-24 17:19:28,599 WARN [HiveServer2-Handler-Pool: Thread-22]:
security.UserGroupInformation (UserGroupInformation.java:doAs(1674)) -
PriviledgedActionException as:hive (auth:SIMPLE)
cause:org.apache.hive.service.cli.HiveSQLException: Error while processing
statement: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
2015-07-24 17:19:28,600 WARN [HiveServer2-Handler-Pool: Thread-22]:
thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(496)) - Error
executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:146)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:173)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:379)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccess
[jira] [Created] (HIVE-6294) Support for CREATE INDEX IF NOT EXISTS
David Morel created HIVE-6294: - Summary: Support for CREATE INDEX IF NOT EXISTS Key: HIVE-6294 URL: https://issues.apache.org/jira/browse/HIVE-6294 Project: Hive Issue Type: Wish Components: Database/Schema Affects Versions: 0.10.0 Reporter: David Morel It would be nice if it were possible to skip index creation on a table if the index already exists. This would simplify greatly some Oozie workflows that I have. It seems generally (not only for indexes) Hive supports more cases of IF EXISTS than IF NOT EXISTS. However, for automation, the latter is often quite useful. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-3869) SELECT foo, NULL UNION ALL SELECT bar, baz fails
David Morel created HIVE-3869:
-
Summary: SELECT foo, NULL UNION ALL SELECT bar, baz fails
Key: HIVE-3869
URL: https://issues.apache.org/jira/browse/HIVE-3869
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Morel
In order to avoid the curse of the last reducer by using a left outer join
where most joined rows woudl be NULLs, I rewrote the query as:
{code}
SELECT * FROM (
SELECT
A.user_id id,
B.created
FROM (
SELECT DISTINCT user_id
FROM users
) A
JOIN
buyhist B
ON
A.user_id = B.user_id
AND B.created >= '2013-01-01'
UNION ALL
SELECT
DISTINCT(user_id) id,
NULL created
FROM users
) foo;
{code}
The expection thrown is this:
{code}
2013-01-07 17:00:01,081 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector.toString(StructObjectInspector.java:60)
at java.lang.String.valueOf(String.java:2826)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:110)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:427)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 22 more
{code}
The
"org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector.toString(StructObjectInspector.java:60)"
caught my attention, so I replaced NULL by an empty string:
{code}
...
UNION ALL
SELECT
DISTINCT(user_id) id,
'' created
{code}
Shouldn't the query parser accept the form using NULL, or at least output a
message before the job is sent to the jobtracker?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3870) SELECT foo, NULL UNION ALL SELECT bar, baz fails
David Morel created HIVE-3870:
-
Summary: SELECT foo, NULL UNION ALL SELECT bar, baz fails
Key: HIVE-3870
URL: https://issues.apache.org/jira/browse/HIVE-3870
Project: Hive
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: David Morel
In order to avoid the curse of the last reducer by using a left outer join
where most joined rows woudl be NULLs, I rewrote the query as:
{code}
SELECT * FROM (
SELECT
A.user_id id,
B.created
FROM (
SELECT DISTINCT user_id
FROM users
) A
JOIN
buyhist B
ON
A.user_id = B.user_id
AND B.created >= '2013-01-01'
UNION ALL
SELECT
DISTINCT(user_id) id,
NULL created
FROM users
) foo;
{code}
The expection thrown is this:
{code}
2013-01-07 17:00:01,081 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector.toString(StructObjectInspector.java:60)
at java.lang.String.valueOf(String.java:2826)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:110)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:427)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 22 more
{code}
The
"org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector.toString(StructObjectInspector.java:60)"
caught my attention, so I replaced NULL by an empty string:
{code}
...
UNION ALL
SELECT
DISTINCT(user_id) id,
'' created
{code}
Shouldn't the query parser accept the form using NULL, or at least output a
message before the job is sent to the jobtracker?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
