[jira] [Commented] (HIVE-9480) Build UDF TRUNC to implement FIRST_DAY as compared with LAST_DAY
[
https://issues.apache.org/jira/browse/HIVE-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366640#comment-14366640
]
Alexander Pivovarov commented on HIVE-9480:
---
Added the function description to the wiki
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
Build UDF TRUNC to implement FIRST_DAY as compared with LAST_DAY
Key: HIVE-9480
URL: https://issues.apache.org/jira/browse/HIVE-9480
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
Labels: TODOC1.2
Fix For: 1.2.0
Attachments: HIVE-9480.1.patch, HIVE-9480.3.patch, HIVE-9480.4.patch,
HIVE-9480.5.patch, HIVE-9480.6.patch, HIVE-9480.7.patch, HIVE-9480.8.patch,
HIVE-9480.9.patch
Hive already supports LAST_DAY UDF, in some cases, FIRST_DAY is necessary to
do date/timestamp related computation. This JIRA is to track such an
implementation. Choose to impl TRUNC, a more standard way to get the first
day of a a month, e.g., SELECT TRUNC('2009-12-12', 'MM'); will return
2009-12-01, SELECT TRUNC('2009-12-12', 'YEAR'); will return 2009-01-01.
BTW, this TRUNC is not as feature complete as aligned with Oracle one. only
'MM' and 'YEAR' are supported as format, however, it's a base to add on other
formats.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9975) Renaming a nonexisting partition should not throw out NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366583#comment-14366583
]
Szehon Ho commented on HIVE-9975:
-
+1, thanks Chaoyu
Renaming a nonexisting partition should not throw out NullPointerException
--
Key: HIVE-9975
URL: https://issues.apache.org/jira/browse/HIVE-9975
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9975.1.patch, HIVE-9975.2.patch, HIVE-9975.patch
Renaming a nonexisting partition should not throw out NullPointerException.
create table testpart (col1 int, col2 string, col3 string) partitioned by
(part string);
alter table testpart partition (part = 'nonexisting') rename to partition
(part = 'existing');
we get NPE like following:
{code}
15/03/16 10:16:11 ERROR exec.DDLTask: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.DDLTask.renamePartition(DDLTask.java:944)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:350)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1642)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1402)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1187)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9984) JoinReorder's getOutputSize is exponential
[
https://issues.apache.org/jira/browse/HIVE-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364731#comment-14364731
]
Hive QA commented on HIVE-9984:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704937/HIVE-9984.2.patch
{color:green}SUCCESS:{color} +1 7769 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3050/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3050/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3050/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704937 - PreCommit-HIVE-TRUNK-Build
JoinReorder's getOutputSize is exponential
--
Key: HIVE-9984
URL: https://issues.apache.org/jira/browse/HIVE-9984
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-9984.1.patch, HIVE-9984.2.patch
Found by [~mmokhtar]. Causes major issues in large plans (50+ joins). Simple
fix would be to memoize the recursion. There should also be a flag to switch
this opt off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9993) Retrying task could use cached bad operators [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365946#comment-14365946 ] Szehon Ho commented on HIVE-9993: - +1, good we caught it Retrying task could use cached bad operators [Spark Branch] --- Key: HIVE-9993 URL: https://issues.apache.org/jira/browse/HIVE-9993 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: spark-branch Attachments: HIVE-9993.1-spark.patch For a Spark task, it could be retried on the same executor in case some failures. In retrying, the cache task could be used. Since the operators in the task are already initialized, they won't be initialized again. The partial data in these operators could lead to wrong final results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9986) LLAP: EOFException in reader
[
https://issues.apache.org/jira/browse/HIVE-9986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366040#comment-14366040
]
Sergey Shelukhin commented on HIVE-9986:
May be fixed in HIVE-9978, unless it's a different EOF. I'll take a look at the
logs
LLAP: EOFException in reader
Key: HIVE-9986
URL: https://issues.apache.org/jira/browse/HIVE-9986
Project: Hive
Issue Type: Sub-task
Reporter: Gopal V
Assignee: Sergey Shelukhin
Fix For: llap
From HIVE-9979
{noformat}
2015-03-16 10:20:51,439
[pool-2-thread-3(container_1_1141_01_000192_gopal_20150316102020_c8c92488-6a61-401e-8298-401dace286dc:1_Map
1_191_0)] INFO org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl: Getting
data for column 9 RG 112 stream DATA at 62278935, 1057137 index position 0:
compressed [62614934, 63139228)
2015-03-16 10:20:51,439
[pool-2-thread-6(container_1_1141_01_000211_gopal_20150316102020_c8c92488-6a61-401e-8298-401dace286dc:1_Map
1_210_0)] INFO org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl: Getting
stripe-level stream [LENGTH, kind: DICTIONARY_V2
dictionarySize: 3
] for column 9 RG 91 at 64139927, 5
...
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.readDirect(RecordReaderUtils.java:286)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:266)
at
org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:234)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:280)
at
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:44)
at
org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
... 4 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-9729) LLAP: design and implement proper metadata cache
[ https://issues.apache.org/jira/browse/HIVE-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-9729. Resolution: Fixed LLAP: design and implement proper metadata cache Key: HIVE-9729 URL: https://issues.apache.org/jira/browse/HIVE-9729 Project: Hive Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: llap Simple approach: add external priorities to data cache, read metadata parts of orc file into it. Advantage: simple; consistent management (no need to coordinate sizes and eviction between data and metadata caches, etc); disadvantage - have to decode every time. Maybe add decoded metadata cache on top - fixed size, small and opportunistic? Or some other approach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9996) [CBO] Generate appropriate join operator as per algorithm selected by CBO
[ https://issues.apache.org/jira/browse/HIVE-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-9996: --- Attachment: HIVE-9996.cbo.patch [CBO] Generate appropriate join operator as per algorithm selected by CBO - Key: HIVE-9996 URL: https://issues.apache.org/jira/browse/HIVE-9996 Project: Hive Issue Type: Task Components: CBO, Query Planning Affects Versions: cbo-branch Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9996.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9989) hive on tez group by NPE
[ https://issues.apache.org/jira/browse/HIVE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-9989: - Attachment: test.sql errorlog hive on tez group by NPE Key: HIVE-9989 URL: https://issues.apache.org/jira/browse/HIVE-9989 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Environment: hive 0.14 tez 0.53 Reporter: tangjunjie Attachments: errorlog, test.sql set hive.execution.engine=tez run sql throw NPE when I modify my sql (see my test.sql file line 14) to cast(NULL AS BIGINT).The sql run ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9971) Clean up operator class
[
https://issues.apache.org/jira/browse/HIVE-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364893#comment-14364893
]
Hive QA commented on HIVE-9971:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705002/HIVE-9971.4.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7769 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_bmj_schema_evolution
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_main
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3052/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3052/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3052/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705002 - PreCommit-HIVE-TRUNK-Build
Clean up operator class
---
Key: HIVE-9971
URL: https://issues.apache.org/jira/browse/HIVE-9971
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Attachments: HIVE-9971.1.patch, HIVE-9971.2.patch, HIVE-9971.3.patch,
HIVE-9971.4.patch
This is mostly cleanup although it does enhance the pipeline in one respect.
It introduces asyn init for operators and uses it for hash table loading
where desired.
There's a bunch of weird code associated with the operator class:
- initialize isn't recursive, rather initializeOp is supposed to call
initializeChildren. That has led to bugs in the past.
- setExecContext and passExecContext. Both are recursive, but passExecContext
calls setExecContext and then recurses again. Boo.
- lots of (getChildren() != null) although that can't happen anymore
- TezCacheAccess is a hack. We should just leave init of inputs up to the
operator that needs it.
- Need some sanity checks that make sure that operators were all initialized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9989) hive on tez group by NPE
[ https://issues.apache.org/jira/browse/HIVE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-9989: - Attachment: errorlog.TXT hive on tez group by NPE Key: HIVE-9989 URL: https://issues.apache.org/jira/browse/HIVE-9989 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Environment: hive 0.14 tez 0.53 Reporter: tangjunjie Attachments: errorlog.TXT, test.sql set hive.execution.engine=tez run sql throw NPE when I modify my sql (see my test.sql file line 14) to cast(NULL AS BIGINT).The sql run ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9989) hive on tez group by NPE
[ https://issues.apache.org/jira/browse/HIVE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-9989: - Attachment: (was: errorlog) hive on tez group by NPE Key: HIVE-9989 URL: https://issues.apache.org/jira/browse/HIVE-9989 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Environment: hive 0.14 tez 0.53 Reporter: tangjunjie Attachments: errorlog.TXT, test.sql set hive.execution.engine=tez run sql throw NPE when I modify my sql (see my test.sql file line 14) to cast(NULL AS BIGINT).The sql run ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364857#comment-14364857
]
Pavan Srinivas commented on HIVE-9543:
--
Hey [~brocknoland], as [~lijy83] pointed out, when ever there is a
MysqlException, a duplicate entry will be created. The problem here is with
https://db.apache.org/jdo/api20/apidocs/javax/jdo/Query.html#execute%28%29
execute() function of Query.java. execute() does not throw any exceptions.
Some reproducible steps:
1) Make sure metastore db is reachablerunning.
2) Enable debugger and stop at a breakpoint inside checkSchema().
3) Stop the metastore db or make it unreachable.
4) Continue on the debugger, you would see execute call of getMSchemaVersion()
does not throw exceptions of DB reachability. It just returns an empty array as
if there are no entries in the table. Thus making the getMSchemaVersion() throw
NoSuchObjectException.
This issue happens when we have a flacky network connection to DB.
MetaException(message:Metastore contains multiple versions)
---
Key: HIVE-9543
URL: https://issues.apache.org/jira/browse/HIVE-9543
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.13.1
Reporter: Junyong Li
When i run bin/hive command, i got the following exception:
{noformat}
Logging initialized using configuration in
jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
Exception in thread main java.lang.RuntimeException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:62)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
... 7 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
... 12 more
Caused by: MetaException(message:Metastore contains multiple versions)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
at
org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:356)
at
[jira] [Commented] (HIVE-9985) Vectorization: NPE for added columns in ORC non-partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364800#comment-14364800
]
Hive QA commented on HIVE-9985:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704929/HIVE-9985.01.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7770 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partitioned_date_time
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partitioned_date_time
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_part
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_part
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3051/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3051/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3051/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704929 - PreCommit-HIVE-TRUNK-Build
Vectorization: NPE for added columns in ORC non-partitioned tables
--
Key: HIVE-9985
URL: https://issues.apache.org/jira/browse/HIVE-9985
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 0.14.0
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical
Attachments: HIVE-9985.01.patch
If you add STRING columns to a non-partitioned table (ORC format) and try to
read the added STRING column using vectorization, you will get a
NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9989) hive on tez group by NPE
[ https://issues.apache.org/jira/browse/HIVE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tangjunjie updated HIVE-9989: - Description: set hive.execution.engine=tez run sql throw NPE when I modify my sql (see my test.sql file line 14) to cast(NULL AS BIGINT).The sql run ok. was:set hive.execution.engine=tez run sql throw NPE hive on tez group by NPE Key: HIVE-9989 URL: https://issues.apache.org/jira/browse/HIVE-9989 Project: Hive Issue Type: Bug Affects Versions: 0.14.0 Environment: hive 0.14 tez 0.53 Reporter: tangjunjie set hive.execution.engine=tez run sql throw NPE when I modify my sql (see my test.sql file line 14) to cast(NULL AS BIGINT).The sql run ok. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9792) Support interval type in expressions/predicates
[
https://issues.apache.org/jira/browse/HIVE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365600#comment-14365600
]
Ashutosh Chauhan commented on HIVE-9792:
Patch mostly looks good.
Left few minor comments on RB.
Thanks for excellent system unit tests!
Support interval type in expressions/predicates
Key: HIVE-9792
URL: https://issues.apache.org/jira/browse/HIVE-9792
Project: Hive
Issue Type: Sub-task
Components: Types
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-9792.1.patch, HIVE-9792.2.patch, HIVE-9792.3.patch,
HIVE-9792.4.patch, HIVE-9792.5.patch, HIVE-9792.6.patch
Provide partial support for the interval year-month/interval day-time types
in Hive. Intervals will be usable in expressions/predicates/joins:
{noformat}
select birthdate + interval '30-0' year to month as thirtieth_birthday
from table
where (current_timestamp - ts1 interval '3 0:0:0' day to second)
{noformat}
This stops short of adding making the interval types usable as a storable
column type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365779#comment-14365779
]
Hive QA commented on HIVE-9991:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705104/HIVE-9991.1.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7770 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_encryption_unencrypted_nonhdfs_external_tables
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3058/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3058/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3058/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705104 - PreCommit-HIVE-TRUNK-Build
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch, HIVE-9991.2.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[ https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365832#comment-14365832 ] Xuefu Zhang commented on HIVE-9934: --- [~prasadm], I think lacking @Test seems fine in this case, as the class is extended from TestCase. I also saw the added test case was run in previous test result. Thus, patch #3 is good as far as I can see. Let me know if you see differently. Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password -- Key: HIVE-9934 URL: https://issues.apache.org/jira/browse/HIVE-9934 Project: Hive Issue Type: Bug Components: Security Affects Versions: 1.1.0 Reporter: Chao Assignee: Chao Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch, HIVE-9934.3.patch, HIVE-9934.4.patch Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password. See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html “If you supply an empty string, an empty byte/char array, or null to the Context.SECURITY_CREDENTIALS environment property, then the authentication mechanism will be none. This is because the LDAP requires the password to be nonempty for simple authentication. The protocol automatically converts the authentication to none if a password is not supplied.” Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a NamingException being thrown during creation of initial context, it does not fail when the context result is an “unauthenticated” positive response from the LDAP server. The end result is, one can authenticate with HiveServer2 using the LdapAuthenticationProviderImpl with only a user name and an empty password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365606#comment-14365606
]
Xuefu Zhang commented on HIVE-9991:
---
+1 pending on test
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9975) Renaming a nonexisting partition should not throw out NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-9975:
--
Attachment: HIVE-9975.1.patch
Fixed alter_rename_partition_failure.q negative test failure. The error message
should be:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Renamed partition does not exist in
default.alter_rename_partition
instead of the previous
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. null
Renaming a nonexisting partition should not throw out NullPointerException
--
Key: HIVE-9975
URL: https://issues.apache.org/jira/browse/HIVE-9975
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9975.1.patch, HIVE-9975.patch
Renaming a nonexisting partition should not throw out NullPointerException.
create table testpart (col1 int, col2 string, col3 string) partitioned by
(part string);
alter table testpart partition (part = 'nonexisting') rename to partition
(part = 'existing');
we get NPE like following:
{code}
15/03/16 10:16:11 ERROR exec.DDLTask: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.DDLTask.renamePartition(DDLTask.java:944)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:350)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1642)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1402)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1187)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9993) Retrying task could use cached bad operators [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-9993: -- Attachment: HIVE-9993.1-spark.patch Retrying task could use cached bad operators [Spark Branch] --- Key: HIVE-9993 URL: https://issues.apache.org/jira/browse/HIVE-9993 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: spark-branch Attachments: HIVE-9993.1-spark.patch For a Spark task, it could be retried on the same executor in case some failures. In retrying, the cache task could be used. Since the operators in the task are already initialized, they won't be initialized again. The partial data in these operators could lead to wrong final results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9978) LLAP: OrcColumnVectorProducer should handle reading isPresent columns only
[
https://issues.apache.org/jira/browse/HIVE-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-9978:
---
Attachment: HIVE-9978.patch
LLAP: OrcColumnVectorProducer should handle reading isPresent columns only
--
Key: HIVE-9978
URL: https://issues.apache.org/jira/browse/HIVE-9978
Project: Hive
Issue Type: Sub-task
Components: File Formats
Affects Versions: llap
Reporter: Gopal V
Assignee: Sergey Shelukhin
Attachments: HIVE-9978.patch
LlapInputFormat does not understand the difference between empty columns list
and null columns list.
The empty columns list indicates no columns read except the root struct
isPresent column, while the null columns list indicates that all columns are
being read.
{code}
select count(1) from store_sales join date_dim on ss_sold_date_sk = d_date_sk
where d_date = '1998-01-01';
...
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.decode.OrcColumnVectorProducer.createReadPipeline(OrcColumnVectorProducer.java:72)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.startRead(LlapInputFormat.java:181)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:99)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[ https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9934: -- Attachment: HIVE-9934.4.patch Update the patch, adding @Test annotation. Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password -- Key: HIVE-9934 URL: https://issues.apache.org/jira/browse/HIVE-9934 Project: Hive Issue Type: Bug Components: Security Affects Versions: 1.1.0 Reporter: Chao Assignee: Chao Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch, HIVE-9934.3.patch, HIVE-9934.4.patch Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password. See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html “If you supply an empty string, an empty byte/char array, or null to the Context.SECURITY_CREDENTIALS environment property, then the authentication mechanism will be none. This is because the LDAP requires the password to be nonempty for simple authentication. The protocol automatically converts the authentication to none if a password is not supplied.” Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a NamingException being thrown during creation of initial context, it does not fail when the context result is an “unauthenticated” positive response from the LDAP server. The end result is, one can authenticate with HiveServer2 using the LdapAuthenticationProviderImpl with only a user name and an empty password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9828) Semantic analyzer does not capture view parent entity for tables referred in view with union all
[
https://issues.apache.org/jira/browse/HIVE-9828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasad Mujumdar updated HIVE-9828:
--
Attachment: HIVE-9828.1-npf.patch
Reattaching for pre-commit run
Semantic analyzer does not capture view parent entity for tables referred in
view with union all
-
Key: HIVE-9828
URL: https://issues.apache.org/jira/browse/HIVE-9828
Project: Hive
Issue Type: Bug
Components: Parser
Affects Versions: 1.1.0
Reporter: Prasad Mujumdar
Fix For: 1.2.0
Attachments: HIVE-9828.1-npf.patch, HIVE-9828.1-npf.patch
Hive compiler adds tables used in a view definition in the input entity list,
with the view as parent entity for the table.
In case of a view with union all query, this is not being done property. For
example,
{noformat}
create view view1 as select t.id from (select tab1.id from db.tab1 union all
select tab2.id from db.tab2 ) t;
{noformat}
This query will capture tab1 and tab2 as read entity without view1 as parent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9991:
--
Attachment: HIVE-9991.2.patch
New patch with just a small changes:
- Change equals() to equalsIgnoreCase()
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch, HIVE-9991.2.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9964) CBO (Calcite Return Path): Traits propagation for Aggregate operator [CBO Branch]
[ https://issues.apache.org/jira/browse/HIVE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365824#comment-14365824 ] Jesus Camacho Rodriguez commented on HIVE-9964: --- [~ashutoshc], I think this patch could go in so Mostafa gets the correct trait set propagation. CBO (Calcite Return Path): Traits propagation for Aggregate operator [CBO Branch] - Key: HIVE-9964 URL: https://issues.apache.org/jira/browse/HIVE-9964 Project: Hive Issue Type: Sub-task Components: CBO Affects Versions: cbo-branch Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez Fix For: cbo-branch Attachments: HIVE-9964.cbo.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9992) Operator Precedence is incorrect. bitwise XOR should be after minus/plus
[
https://issues.apache.org/jira/browse/HIVE-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov resolved HIVE-9992.
---
Resolution: Not a Problem
I checked MySQL doc again. Bitwise XOR ^ has higher precedence than -/+ in
MySQL.
http://dev.mysql.com/doc/refman/5.0/en/operator-precedence.html
Hive implemented similar way.
Operator Precedence is incorrect. bitwise XOR should be after minus/plus
Key: HIVE-9992
URL: https://issues.apache.org/jira/browse/HIVE-9992
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
bitwise XOR ^ has higher precedence than -/+ in hive
But in java bitwise XOR has lower precedence
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html
{code}
// now
select 4 - 2 ^ 8 - 7;
-13
// should be
select 4 - 2 ^ 8 - 7;
3
{code}
need to fix
IdentifiersParser.g BITWISEXOR
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9975) Renaming a nonexisting partition should not throw out NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365598#comment-14365598
]
Hive QA commented on HIVE-9975:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704962/HIVE-9975.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7769 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_rename_partition_failure
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3057/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3057/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3057/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704962 - PreCommit-HIVE-TRUNK-Build
Renaming a nonexisting partition should not throw out NullPointerException
--
Key: HIVE-9975
URL: https://issues.apache.org/jira/browse/HIVE-9975
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9975.patch
Renaming a nonexisting partition should not throw out NullPointerException.
create table testpart (col1 int, col2 string, col3 string) partitioned by
(part string);
alter table testpart partition (part = 'nonexisting') rename to partition
(part = 'existing');
we get NPE like following:
{code}
15/03/16 10:16:11 ERROR exec.DDLTask: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.DDLTask.renamePartition(DDLTask.java:944)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:350)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1642)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1402)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1187)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9792) Support interval type in expressions/predicates
[
https://issues.apache.org/jira/browse/HIVE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365770#comment-14365770
]
Ashutosh Chauhan commented on HIVE-9792:
+1
Support interval type in expressions/predicates
Key: HIVE-9792
URL: https://issues.apache.org/jira/browse/HIVE-9792
Project: Hive
Issue Type: Sub-task
Components: Types
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-9792.1.patch, HIVE-9792.2.patch, HIVE-9792.3.patch,
HIVE-9792.4.patch, HIVE-9792.5.patch, HIVE-9792.6.patch, HIVE-9792.7.patch
Provide partial support for the interval year-month/interval day-time types
in Hive. Intervals will be usable in expressions/predicates/joins:
{noformat}
select birthdate + interval '30-0' year to month as thirtieth_birthday
from table
where (current_timestamp - ts1 interval '3 0:0:0' day to second)
{noformat}
This stops short of adding making the interval types usable as a storable
column type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without
[
https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9934:
--
Comment: was deleted
(was:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705130/HIVE-9934.4.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3059/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3059/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3059/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.0 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.3.1 from the
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0 from
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.3.1 from the
shaded jar.
[INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1 from the
shaded jar.
[INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar.
[INFO] Excluding com.codahale.metrics:metrics-core:jar:3.0.0 from the shaded
jar.
[INFO] Excluding com.codahale.metrics:metrics-jvm:jar:3.0.0 from the shaded jar.
[INFO] Excluding com.codahale.metrics:metrics-json:jar:3.0.0 from the shaded
jar.
[INFO] Excluding com.codahale.metrics:metrics-graphite:jar:3.0.0 from the
shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-client:jar:0.5.0 from the shaded
jar.
[INFO] Excluding org.tachyonproject:tachyon:jar:0.5.0 from the shaded jar.
[INFO] Excluding org.spark-project:pyrolite:jar:2.0.1 from the shaded jar.
[INFO] Excluding net.sf.py4j:py4j:jar:0.8.2.1 from the shaded jar.
[INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.6.0 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.14 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-json:jar:1.14 from the shaded jar.
[INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar.
[INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.14 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar.
[INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.6.0 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar.
[INFO] Excluding org.htrace:htrace-core:jar:3.0.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-archives:jar:2.6.0 from the shaded
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0 from
the shaded jar.
[INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from the
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.0 from the
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.6.0 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding
[jira] [Updated] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[ https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9934: -- Attachment: (was: HIVE-9934.4.patch) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password -- Key: HIVE-9934 URL: https://issues.apache.org/jira/browse/HIVE-9934 Project: Hive Issue Type: Bug Components: Security Affects Versions: 1.1.0 Reporter: Chao Assignee: Chao Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch, HIVE-9934.3.patch Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password. See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html “If you supply an empty string, an empty byte/char array, or null to the Context.SECURITY_CREDENTIALS environment property, then the authentication mechanism will be none. This is because the LDAP requires the password to be nonempty for simple authentication. The protocol automatically converts the authentication to none if a password is not supplied.” Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a NamingException being thrown during creation of initial context, it does not fail when the context result is an “unauthenticated” positive response from the LDAP server. The end result is, one can authenticate with HiveServer2 using the LdapAuthenticationProviderImpl with only a user name and an empty password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9859) Create bitwise left/right shift UDFs
[
https://issues.apache.org/jira/browse/HIVE-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9859:
--
Summary: Create bitwise left/right shift UDFs (was: Create bitwise shift
left/right UDFs)
Create bitwise left/right shift UDFs
Key: HIVE-9859
URL: https://issues.apache.org/jira/browse/HIVE-9859
Project: Hive
Issue Type: Improvement
Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Attachments: HIVE-9859.1.patch
Signature:
a b
a b
a b
For example:
{code}
select 1 4, 8 2, 8 2;
OK
16 2 2
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366556#comment-14366556
]
Hive QA commented on HIVE-9991:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705207/HIVE-9991.4.patch
{color:green}SUCCESS:{color} +1 7771 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3065/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3065/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3065/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705207 - PreCommit-HIVE-TRUNK-Build
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch, HIVE-9991.2.patch, HIVE-9991.3.patch,
HIVE-9991.4.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9999) LLAP: Handle task rejection from daemons in the AM
[ https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-: - Attachment: HIVE-.1.patch LLAP: Handle task rejection from daemons in the AM -- Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10000) 10000 whoooohooo
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366335#comment-14366335 ] Sergey Shelukhin commented on HIVE-1: - Here I was hoping to file some aspirational LLAP JIRA under than number 1 whhooo Key: HIVE-1 URL: https://issues.apache.org/jira/browse/HIVE-1 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-10000) 10000 whoooohooo
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366335#comment-14366335 ] Sergey Shelukhin edited comment on HIVE-1 at 3/17/15 11:56 PM: --- Here I was hoping to file some aspirational LLAP JIRA under this number was (Author: sershe): Here I was hoping to file some aspirational LLAP JIRA under than number 1 whhooo Key: HIVE-1 URL: https://issues.apache.org/jira/browse/HIVE-1 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5016) Local mode FileNotFoundException: File does not exist
[ https://issues.apache.org/jira/browse/HIVE-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366243#comment-14366243 ] Jian Fang commented on HIVE-5016: - Hi Abin, seems this issue still exits in the latest hadoop 2.6.0 code. Could you please create a Jira to hadoop? Thanks. Local mode FileNotFoundException: File does not exist - Key: HIVE-5016 URL: https://issues.apache.org/jira/browse/HIVE-5016 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Environment: Centos 6.3 (final) Hadoop 2.0.2-alpha Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Hive libs: ls -1 lib/ antlr-2.7.7.jar antlr-runtime-3.0.1.jar avro-1.7.1.jar avro-mapred-1.7.1.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-compress-1.4.1.jar commons-configuration-1.6.jar commons-dbcp-1.4.jar commons-lang-2.4.jar commons-logging-1.0.4.jar commons-logging-api-1.0.4.jar commons-pool-1.5.4.jar datanucleus-connectionpool-2.0.3.jar datanucleus-core-2.0.3.jar datanucleus-enhancer-2.0.3.jar datanucleus-rdbms-2.0.3.jar derby-10.4.2.0.jar guava-r09.jar hbase-0.92.0.jar hbase-0.92.0-tests.jar hive-builtins-0.10.0.jar hive-cli-0.10.0.jar hive-common-0.10.0.jar hive-contrib-0.10.0.jar hive-exec-0.10.0.jar hive-hbase-handler-0.10.0.jar hive-hwi-0.10.0.jar hive-hwi-0.10.0.war hive-jdbc-0.10.0.jar hive-metastore-0.10.0.jar hive-pdk-0.10.0.jar hive-serde-0.10.0.jar hive-service-0.10.0.jar hive-shims-0.10.0.jar jackson-core-asl-1.8.8.jar jackson-jaxrs-1.8.8.jar jackson-mapper-asl-1.8.8.jar jackson-xc-1.8.8.jar JavaEWAH-0.3.2.jar javolution-5.5.1.jar jdo2-api-2.3-ec.jar jetty-6.1.26.jar jetty-util-6.1.26.jar jline-0.9.94.jar json-20090211.jar libfb303-0.9.0.jar libthrift-0.9.0.jar log4j-1.2.16.jar php py servlet-api-2.5-20081211.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar sqlline-1_0_2.jar stringtemplate-3.1-b1.jar xz-1.0.jar zookeeper-3.4.3.jar Reporter: Abin Shahab Priority: Critical Attachments: HIVE-5016.patch, HIVE-5016.patch Hive jobs in local mode fail with the error posted below. The jar file that's not being found exists and has the following access: ls -l hive-0.10.0/lib/hive-builtins-0.10.0.jar rw-rw-r-- 1 ashahab ashahab 3914 Dec 18 2012 hive-0.10.0/lib/hive-builtins-0.10.0.jar Steps to reproduce: hive set hive.exec.mode.local.auto=true; hive set hive.exec.mode.local.auto; hive.exec.mode.local.auto=true hive select count(*) from abin_test_table; Automatically selecting local only mode for query Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number 13/08/06 21:37:11 WARN conf.Configuration: file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 13/08/06 21:37:11 WARN conf.Configuration: file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/ashahab/ashahab_20130806213737_7d26b796-5f55-44ca-a755-8898153d963b.log java.io.FileNotFoundException: File does not exist: /home/ashahab/dev/hive-0.10.0/lib/hive-builtins-0.10.0.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:252) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:290) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:361) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at
[jira] [Updated] (HIVE-9919) upgrade scripts don't work on some auto-created DBs due to absence of tables
[ https://issues.apache.org/jira/browse/HIVE-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9919: --- Attachment: HIVE-9919.01.patch Updated the patch to add all the indexes/etc. from all the respective schemas, in case future scripts decide to drop them (which they do) upgrade scripts don't work on some auto-created DBs due to absence of tables Key: HIVE-9919 URL: https://issues.apache.org/jira/browse/HIVE-9919 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9919.01.patch, HIVE-9919.patch DataNucleus in its infinite wisdom doesn't create all tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366213#comment-14366213
]
Xuefu Zhang commented on HIVE-9991:
---
[~spena], it seems the above failed test has a result diff. You might need to
regenerate the test output.
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch, HIVE-9991.2.patch, HIVE-9991.3.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10000) 10000 whoooohooo
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366337#comment-14366337 ] Gunther Hagleitner commented on HIVE-1: --- [~sershe] what do you have in mind. I can s till change the title :-P 1 whhooo Key: HIVE-1 URL: https://issues.apache.org/jira/browse/HIVE-1 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9997) minor tweaks for bytes mapjoin hash table
[ https://issues.apache.org/jira/browse/HIVE-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9997: --- Attachment: HIVE-9997.patch first attempt; need to see HiveQA minor tweaks for bytes mapjoin hash table - Key: HIVE-9997 URL: https://issues.apache.org/jira/browse/HIVE-9997 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9997.patch From HIVE-7617 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366209#comment-14366209 ] Yongzhi Chen commented on HIVE-7018: Here is my plan: I will find a way to check if the two tables have LINK_TARGE_ID column or not, if have it, I will drop the column. Is that a safe plan? Thanks Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others - Key: HIVE-7018 URL: https://issues.apache.org/jira/browse/HIVE-7018 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Yongzhi Chen Attachments: HIVE-7018.1.patch It appears that at least postgres and oracle do not have the LINK_TARGET_ID column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9997) minor tweaks for bytes mapjoin hash table
[ https://issues.apache.org/jira/browse/HIVE-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366272#comment-14366272 ] Sergey Shelukhin commented on HIVE-9997: https://reviews.apache.org/r/32178 minor tweaks for bytes mapjoin hash table - Key: HIVE-9997 URL: https://issues.apache.org/jira/browse/HIVE-9997 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9997.patch From HIVE-7617 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10000) 10000 whoooohooo
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366371#comment-14366371 ] Sergey Shelukhin commented on HIVE-1: - Define APIs? :) [~sseth] 1 whhooo Key: HIVE-1 URL: https://issues.apache.org/jira/browse/HIVE-1 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9999) LLAP: Handle task rejection from daemons in the AM
[ https://issues.apache.org/jira/browse/HIVE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved HIVE-. -- Resolution: Fixed LLAP: Handle task rejection from daemons in the AM -- Key: HIVE- URL: https://issues.apache.org/jira/browse/HIVE- Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap Attachments: HIVE-.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9998) Vectorization support for interval types
[ https://issues.apache.org/jira/browse/HIVE-9998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-9998: - Attachment: HIVE-9998.1.patch Initial patch, dependent on HIVE-9792. This also fixes some vectorized comparisons for Date, when one of the left or right side is a constant expression. Vectorization support for interval types Key: HIVE-9998 URL: https://issues.apache.org/jira/browse/HIVE-9998 Project: Hive Issue Type: Sub-task Components: Types, Vectorization Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-9998.1.patch Allow intervals to work during vectorized query execution -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9997) minor tweaks for bytes mapjoin hash table
[
https://issues.apache.org/jira/browse/HIVE-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366340#comment-14366340
]
Mostafa Mokhtar commented on HIVE-9997:
---
[~sershe]
Can these be addressed as well?
private int getOffset(long offset) {
return (int)(offset % wbSize);
}
private int getBufferIndex(long offset) {
return (int)(offset / wbSize);
}
Called from BytesBytesMultiHashMap.isSameKey
minor tweaks for bytes mapjoin hash table
-
Key: HIVE-9997
URL: https://issues.apache.org/jira/browse/HIVE-9997
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-9997.patch
From HIVE-7617
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9991:
--
Attachment: HIVE-9991.3.patch
Fixed patch to include the test into TestEncryptedHdfsCliDriver tests.
It also uses pfile:/// as a location instead of file:/// in order to avoid
change ownership errors.
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch, HIVE-9991.2.patch, HIVE-9991.3.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9992) Operator Precedence is incorrect. bitwise XOR should be after minus/plus
[
https://issues.apache.org/jira/browse/HIVE-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9992:
--
Summary: Operator Precedence is incorrect. bitwise XOR should be after
minus/plus (was: Operator Precedence is incorrect. XOR should be after
minus/plus)
Operator Precedence is incorrect. bitwise XOR should be after minus/plus
Key: HIVE-9992
URL: https://issues.apache.org/jira/browse/HIVE-9992
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
XOR ^ has higher precedence than -/+ in hive
But in java and in MySQL XOR has lower precedence
http://dev.mysql.com/doc/refman/5.0/en/operator-precedence.html
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html
{code}
// now
select 4 - 2 ^ 8 - 7;
-13
// should be
select 4 - 2 ^ 8 - 7;
3
{code}
need to fix
IdentifiersParser.g BITWISEXOR
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9992) Operator Precedence is incorrect. bitwise XOR should be after minus/plus
[
https://issues.apache.org/jira/browse/HIVE-9992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Pivovarov updated HIVE-9992:
--
Description:
bitwise XOR ^ has higher precedence than -/+ in hive
But in java bitwise XOR has lower precedence
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html
{code}
// now
select 4 - 2 ^ 8 - 7;
-13
// should be
select 4 - 2 ^ 8 - 7;
3
{code}
need to fix
IdentifiersParser.g BITWISEXOR
was:
XOR ^ has higher precedence than -/+ in hive
But in java and in MySQL XOR has lower precedence
http://dev.mysql.com/doc/refman/5.0/en/operator-precedence.html
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html
{code}
// now
select 4 - 2 ^ 8 - 7;
-13
// should be
select 4 - 2 ^ 8 - 7;
3
{code}
need to fix
IdentifiersParser.g BITWISEXOR
Operator Precedence is incorrect. bitwise XOR should be after minus/plus
Key: HIVE-9992
URL: https://issues.apache.org/jira/browse/HIVE-9992
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 1.2.0
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
bitwise XOR ^ has higher precedence than -/+ in hive
But in java bitwise XOR has lower precedence
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html
{code}
// now
select 4 - 2 ^ 8 - 7;
-13
// should be
select 4 - 2 ^ 8 - 7;
3
{code}
need to fix
IdentifiersParser.g BITWISEXOR
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9792) Support interval type in expressions/predicates
[
https://issues.apache.org/jira/browse/HIVE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-9792:
-
Attachment: HIVE-9792.7.patch
Patch v7 - RB feedback from Ashutosh
Support interval type in expressions/predicates
Key: HIVE-9792
URL: https://issues.apache.org/jira/browse/HIVE-9792
Project: Hive
Issue Type: Sub-task
Components: Types
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-9792.1.patch, HIVE-9792.2.patch, HIVE-9792.3.patch,
HIVE-9792.4.patch, HIVE-9792.5.patch, HIVE-9792.6.patch, HIVE-9792.7.patch
Provide partial support for the interval year-month/interval day-time types
in Hive. Intervals will be usable in expressions/predicates/joins:
{noformat}
select birthdate + interval '30-0' year to month as thirtieth_birthday
from table
where (current_timestamp - ts1 interval '3 0:0:0' day to second)
{noformat}
This stops short of adding making the interval types usable as a storable
column type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[
https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365794#comment-14365794
]
Hive QA commented on HIVE-9934:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705130/HIVE-9934.4.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3059/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3059/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3059/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.0 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.3.1 from the
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0 from
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.3.1 from the
shaded jar.
[INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1 from the
shaded jar.
[INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar.
[INFO] Excluding com.codahale.metrics:metrics-core:jar:3.0.0 from the shaded
jar.
[INFO] Excluding com.codahale.metrics:metrics-jvm:jar:3.0.0 from the shaded jar.
[INFO] Excluding com.codahale.metrics:metrics-json:jar:3.0.0 from the shaded
jar.
[INFO] Excluding com.codahale.metrics:metrics-graphite:jar:3.0.0 from the
shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-client:jar:0.5.0 from the shaded
jar.
[INFO] Excluding org.tachyonproject:tachyon:jar:0.5.0 from the shaded jar.
[INFO] Excluding org.spark-project:pyrolite:jar:2.0.1 from the shaded jar.
[INFO] Excluding net.sf.py4j:py4j:jar:0.8.2.1 from the shaded jar.
[INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.6.0 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.14 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-json:jar:1.14 from the shaded jar.
[INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 from the shaded jar.
[INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.14 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding tomcat:jasper-compiler:jar:5.5.23 from the shaded jar.
[INFO] Excluding tomcat:jasper-runtime:jar:5.5.23 from the shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.6.0 from the shaded jar.
[INFO] Excluding
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the
shaded jar.
[INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar.
[INFO] Excluding org.htrace:htrace-core:jar:3.0.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-archives:jar:2.6.0 from the shaded
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0 from
the shaded jar.
[INFO] Excluding com.google.inject.extensions:guice-servlet:jar:3.0 from the
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.0 from the
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.6.0 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[ https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365839#comment-14365839 ] Prasad Mujumdar commented on HIVE-9934: --- That's fine. The test did get run in the pre-commit run for patch #3. sorry about the noise. +1 Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password -- Key: HIVE-9934 URL: https://issues.apache.org/jira/browse/HIVE-9934 Project: Hive Issue Type: Bug Components: Security Affects Versions: 1.1.0 Reporter: Chao Assignee: Chao Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch, HIVE-9934.3.patch Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password. See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html “If you supply an empty string, an empty byte/char array, or null to the Context.SECURITY_CREDENTIALS environment property, then the authentication mechanism will be none. This is because the LDAP requires the password to be nonempty for simple authentication. The protocol automatically converts the authentication to none if a password is not supplied.” Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a NamingException being thrown during creation of initial context, it does not fail when the context result is an “unauthenticated” positive response from the LDAP server. The end result is, one can authenticate with HiveServer2 using the LdapAuthenticationProviderImpl with only a user name and an empty password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9981) Avoid throwing many exceptions when attempting to create new hdfs encryption shim
[
https://issues.apache.org/jira/browse/HIVE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365920#comment-14365920
]
Sergio Peña commented on HIVE-9981:
---
This test is not related with this patch.
Avoid throwing many exceptions when attempting to create new hdfs encryption
shim
-
Key: HIVE-9981
URL: https://issues.apache.org/jira/browse/HIVE-9981
Project: Hive
Issue Type: Improvement
Components: Encryption
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9981.1.patch, HIVE-9981.2.patch
Hadoop23Shims.createHdfsEncryptionShim() is throwing an exception for verions
lower than 2.6.0 everytime a query is executed.
Exceptions are expensive, so rather than throwing them every time, we can use
this design pattern followed for some other functions in Hadoop23Shims -
{code}
protected static final Method accessMethod;
protected static final Method getPasswordMethod;
static {
Method m = null;
try {
m = FileSystem.class.getMethod(access, Path.class, FsAction.class);
} catch (NoSuchMethodException err) {
// This version of Hadoop does not support FileSystem.access().
}
accessMethod = m;
try {
m = Configuration.class.getMethod(getPassword, String.class);
} catch (NoSuchMethodException err) {
// This version of Hadoop does not support getPassword(), just retrieve
password from conf.
m = null;
}
getPasswordMethod = m;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9971) Clean up operator class
[ https://issues.apache.org/jira/browse/HIVE-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-9971: - Attachment: HIVE-9971.5.patch Clean up operator class --- Key: HIVE-9971 URL: https://issues.apache.org/jira/browse/HIVE-9971 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-9971.1.patch, HIVE-9971.2.patch, HIVE-9971.3.patch, HIVE-9971.4.patch, HIVE-9971.5.patch This is mostly cleanup although it does enhance the pipeline in one respect. It introduces asyn init for operators and uses it for hash table loading where desired. There's a bunch of weird code associated with the operator class: - initialize isn't recursive, rather initializeOp is supposed to call initializeChildren. That has led to bugs in the past. - setExecContext and passExecContext. Both are recursive, but passExecContext calls setExecContext and then recurses again. Boo. - lots of (getChildren() != null) although that can't happen anymore - TezCacheAccess is a hack. We should just leave init of inputs up to the operator that needs it. - Need some sanity checks that make sure that operators were all initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9978) LLAP: OrcColumnVectorProducer should handle reading isPresent columns only
[
https://issues.apache.org/jira/browse/HIVE-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HIVE-9978.
Resolution: Fixed
Fix Version/s: llap
committed to branch. Should work now
LLAP: OrcColumnVectorProducer should handle reading isPresent columns only
--
Key: HIVE-9978
URL: https://issues.apache.org/jira/browse/HIVE-9978
Project: Hive
Issue Type: Sub-task
Components: File Formats
Affects Versions: llap
Reporter: Gopal V
Assignee: Sergey Shelukhin
Fix For: llap
Attachments: HIVE-9978.patch
LlapInputFormat does not understand the difference between empty columns list
and null columns list.
The empty columns list indicates no columns read except the root struct
isPresent column, while the null columns list indicates that all columns are
being read.
{code}
select count(1) from store_sales join date_dim on ss_sold_date_sk = d_date_sk
where d_date = '1998-01-01';
...
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.llap.io.decode.OrcColumnVectorProducer.createReadPipeline(OrcColumnVectorProducer.java:72)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.startRead(LlapInputFormat.java:181)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
at
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:99)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 22 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[
https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365037#comment-14365037
]
Chao commented on HIVE-9934:
Found this in log:
{noformat}
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) - 2015-03-17 04:33:32,725 INFO
[pool-1-thread-1] client.RemoteDriver (RemoteDriver.java:call(371)) - Failed to
run job 681ccfbe-bf9f-491c-a2e7-ad513f62d1dc
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) - java.util.concurrent.ExecutionException:
Exception thrown by job
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:311)
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:316)
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:364)
2015-03-17 04:33:32,728 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:317)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
java.lang.Thread.run(Thread.java:744)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) - Caused by: org.apache.spark.SparkException:
Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 0.0 (TID 3,
ip-10-182-56-7.ec2.internal): java.io.FileNotFoundException:
http://10.182.56.7:34690/jars/hive-exec-1.2.0-SNAPSHOT.jar
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.util.Utils$.doFetchFile(Utils.scala:452)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.util.Utils$.fetchFile(Utils.scala:383)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:350)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:347)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
2015-03-17 04:33:32,729 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
2015-03-17 04:33:32,730 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
2015-03-17 04:33:32,730 INFO [stdout-redir-1]: client.SparkClientImpl
(SparkClientImpl.java:run(537)) -at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
2015-03-17
[jira] [Commented] (HIVE-9697) Hive on Spark is not as aggressive as MR on map join [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365072#comment-14365072 ] Chao commented on HIVE-9697: Can we use hive.stats.collect.rawdatasize to control this? I think in default Hive uses rawDataSize, right? And if that is not available (like the issue in HIVE-9560), it will switch to totalSize. Is that right? Hive on Spark is not as aggressive as MR on map join [Spark Branch] --- Key: HIVE-9697 URL: https://issues.apache.org/jira/browse/HIVE-9697 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xin Hao We have a finding during running some Big-Bench cases: when the same small table size threshold is used, Map Join operator will not be generated in Stage Plans for Hive on Spark, while will be generated for Hive on MR. For example, When we run BigBench Q25, the meta info of one input ORC table is as below: totalSize=1748955 (about 1.5M) rawDataSize=123050375 (about 120M) If we use the following parameter settings, set hive.auto.convert.join=true; set hive.mapjoin.smalltable.filesize=2500; set hive.auto.convert.join.noconditionaltask=true; set hive.auto.convert.join.noconditionaltask.size=1; (100M) Map Join will be enabled for Hive on MR mode, while will not be enabled for Hive on Spark. We found that for Hive on MR, the HDFS file size for the table (ContentSummary.getLength(), should approximate the value of ‘totalSize’) will be used to compare with the threshold 100M (smaller than 100M), while for Hive on Spark 'rawDataSize' will be used to compare with the threshold 100M (larger than 100M). That's why MapJoin is not enabled for Hive on Spark for this case. And as a result Hive on Spark will get much lower performance data than Hive on MR for this case. When we set hive.auto.convert.join.noconditionaltask.size=15000; (150M), MapJoin will be enabled for Hive on Spark mode, and Hive on Spark will have similar performance data with Hive on MR by then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9970) Hive on spark
[ https://issues.apache.org/jira/browse/HIVE-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364951#comment-14364951 ] Amithsha commented on HIVE-9970: hive insert into table test values(6,8797); Query ID = hadoop2_20150317163636_4692aa68-56b6-4ea9-ad21-e0f46efe4bfc Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapreduce.job.reduces=number Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask Hive on spark - Key: HIVE-9970 URL: https://issues.apache.org/jira/browse/HIVE-9970 Project: Hive Issue Type: Bug Reporter: Amithsha Hi all, Recently i have configured Spark 1.2.0 and my environment is hadoop 2.6.0 hive 1.1.0 Here i have tried hive on Spark while executing insert into i am getting the following g error. Query ID = hadoop2_20150313162828_8764adad-a8e4-49da-9ef5-35e4ebd6bc63 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapreduce.job.reduces=number Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask Have added the spark-assembly jar in hive lib And also in hive console using the command add jar followed by the steps set spark.home=/opt/spark-1.2.1/; add jar /opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar; set hive.execution.engine=spark; set spark.master=spark://xxx:7077; set spark.eventLog.enabled=true; set spark.executor.memory=512m; set spark.serializer=org.apache.spark.serializer.KryoSerializer; Can anyone suggest Thanks Regards Amithsha -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5341) Link doesn't work. Needs to be updated as mentioned in the Description
[ https://issues.apache.org/jira/browse/HIVE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fahd Khan updated HIVE-5341: The links isn't working. please provide an alternate link to download the files. Link doesn't work. Needs to be updated as mentioned in the Description -- Key: HIVE-5341 URL: https://issues.apache.org/jira/browse/HIVE-5341 Project: Hive Issue Type: Bug Components: Documentation Reporter: Rakesh Chouhan Assignee: Lefty Leverenz Labels: documentation Go to.. Apache HIVE Getting Started Documentation https://cwiki.apache.org/confluence/display/Hive/GettingStarted Under Section ... Simple Example Use Cases MovieLens User Ratings wget http://www.grouplens.org/system/files/ml-data.tar+0.gz The link mentioned as per the document does not work. It needs to be updated to the below URL. http://www.grouplens.org/sites/www.grouplens.org/external_files/data/ml-data.tar.gz I am setting this defect's priority as a Blocker because, user's will not be able to continue their hands on exercises, unless they find the correct URL to download the mentioned file. Referenced from: http://mail-archives.apache.org/mod_mbox/hive-user/201302.mbox/%3c8a0c145b-4db9-4d26-8613-8ca1bd741...@daum.net%3E. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[
https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365024#comment-14365024
]
Hive QA commented on HIVE-9934:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704981/HIVE-9934.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7770 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3053/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3053/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3053/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704981 - PreCommit-HIVE-TRUNK-Build
Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to
degrade the authentication mechanism to none, allowing authentication
without password
--
Key: HIVE-9934
URL: https://issues.apache.org/jira/browse/HIVE-9934
Project: Hive
Issue Type: Bug
Components: Security
Affects Versions: 1.1.0
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch
Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to
degrade the authentication mechanism to none, allowing authentication
without password.
See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html
“If you supply an empty string, an empty byte/char array, or null to the
Context.SECURITY_CREDENTIALS environment property, then the authentication
mechanism will be none. This is because the LDAP requires the password to
be nonempty for simple authentication. The protocol automatically converts
the authentication to none if a password is not supplied.”
Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a
NamingException being thrown during creation of initial context, it does not
fail when the context result is an “unauthenticated” positive response from
the LDAP server. The end result is, one can authenticate with HiveServer2
using the LdapAuthenticationProviderImpl with only a user name and an empty
password.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9977) Compactor not running on partitions after dynamic partitioned insert
[
https://issues.apache.org/jira/browse/HIVE-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365090#comment-14365090
]
Hive QA commented on HIVE-9977:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12704984/HIVE-9977.patch
{color:green}SUCCESS:{color} +1 7774 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3054/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3054/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3054/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12704984 - PreCommit-HIVE-TRUNK-Build
Compactor not running on partitions after dynamic partitioned insert
Key: HIVE-9977
URL: https://issues.apache.org/jira/browse/HIVE-9977
Project: Hive
Issue Type: Bug
Components: Transactions
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Alan Gates
Assignee: Alan Gates
Attachments: HIVE-9977.patch
When an insert, update, or delete is done using dynamic partitioning the lock
is obtained on the table instead of on the individual partitions, since the
partitions are not known at lock acquisition time. The compactor is using
the locks to determine which partitions to check to see if they need
compacted. Since the individual partitions aren't locked they aren't checked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365099#comment-14365099 ] Yongzhi Chen commented on HIVE-7018: [~xuefuz] , [~csun] could you review the change and commit it if looks fine? Thanks Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others - Key: HIVE-7018 URL: https://issues.apache.org/jira/browse/HIVE-7018 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Yongzhi Chen Attachments: HIVE-7018.1.patch It appears that at least postgres and oracle do not have the LINK_TARGET_ID column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9697) Hive on Spark is not as aggressive as MR on map join [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365102#comment-14365102 ] Rui Li commented on HIVE-9697: -- [~csun] - I think MR doesn't use rawDataSize even when it's available. Seems it just uses ContentSummary. Hive on Spark is not as aggressive as MR on map join [Spark Branch] --- Key: HIVE-9697 URL: https://issues.apache.org/jira/browse/HIVE-9697 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xin Hao We have a finding during running some Big-Bench cases: when the same small table size threshold is used, Map Join operator will not be generated in Stage Plans for Hive on Spark, while will be generated for Hive on MR. For example, When we run BigBench Q25, the meta info of one input ORC table is as below: totalSize=1748955 (about 1.5M) rawDataSize=123050375 (about 120M) If we use the following parameter settings, set hive.auto.convert.join=true; set hive.mapjoin.smalltable.filesize=2500; set hive.auto.convert.join.noconditionaltask=true; set hive.auto.convert.join.noconditionaltask.size=1; (100M) Map Join will be enabled for Hive on MR mode, while will not be enabled for Hive on Spark. We found that for Hive on MR, the HDFS file size for the table (ContentSummary.getLength(), should approximate the value of ‘totalSize’) will be used to compare with the threshold 100M (smaller than 100M), while for Hive on Spark 'rawDataSize' will be used to compare with the threshold 100M (larger than 100M). That's why MapJoin is not enabled for Hive on Spark for this case. And as a result Hive on Spark will get much lower performance data than Hive on MR for this case. When we set hive.auto.convert.join.noconditionaltask.size=15000; (150M), MapJoin will be enabled for Hive on Spark mode, and Hive on Spark will have similar performance data with Hive on MR by then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[
https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365134#comment-14365134
]
Xuefu Zhang commented on HIVE-7018:
---
Patch looks fine. However, I don't quite understand why we are also removing
the following:
{code}
- CONSTRAINT `PARTITIONS_FK2` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`),
...
- CONSTRAINT `TBLS_FK2` FOREIGN KEY (`DB_ID`) REFERENCES `DBS` (`DB_ID`),
{code}
This doesn't seem related to LINK_TARGET_ID.
Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but
not others
-
Key: HIVE-7018
URL: https://issues.apache.org/jira/browse/HIVE-7018
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Yongzhi Chen
Attachments: HIVE-7018.1.patch
It appears that at least postgres and oracle do not have the LINK_TARGET_ID
column while mysql does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9664) Hive add jar command should be able to download and add jars from a repository
[
https://issues.apache.org/jira/browse/HIVE-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365100#comment-14365100
]
Hive QA commented on HIVE-9664:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705028/HIVE-9664.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3055/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3055/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3055/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-1.2.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/1.2.0-SNAPSHOT/hive-it-custom-serde-1.2.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/1.2.0-SNAPSHOT/hive-it-custom-serde-1.2.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - HCatalog Unit Tests 1.2.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hcatalog-it-unit
---
[INFO] Deleting
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-hcatalog-it-unit
---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-hcatalog-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-hcatalog-it-unit ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-hcatalog-it-unit ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-hcatalog-it-unit ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @
hive-hcatalog-it-unit ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
[copy] Copying 11 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hcatalog-it-unit ---
[INFO] Compiling 8 source files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/hbase/ManyMiniCluster.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @
hive-hcatalog-it-unit ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-it-unit ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-1.2.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hcatalog-it-unit ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-it-unit ---
[INFO] Building jar:
[jira] [Commented] (HIVE-5341) Link doesn't work. Needs to be updated as mentioned in the Description
[ https://issues.apache.org/jira/browse/HIVE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365112#comment-14365112 ] Fahd Khan commented on HIVE-5341: - you can find the file on the following link as well. Link doesn't work. Needs to be updated as mentioned in the Description -- Key: HIVE-5341 URL: https://issues.apache.org/jira/browse/HIVE-5341 Project: Hive Issue Type: Bug Components: Documentation Reporter: Rakesh Chouhan Assignee: Lefty Leverenz Labels: documentation Go to.. Apache HIVE Getting Started Documentation https://cwiki.apache.org/confluence/display/Hive/GettingStarted Under Section ... Simple Example Use Cases MovieLens User Ratings wget http://www.grouplens.org/system/files/ml-data.tar+0.gz The link mentioned as per the document does not work. It needs to be updated to the below URL. http://www.grouplens.org/sites/www.grouplens.org/external_files/data/ml-data.tar.gz I am setting this defect's priority as a Blocker because, user's will not be able to continue their hands on exercises, unless they find the correct URL to download the mentioned file. Referenced from: http://mail-archives.apache.org/mod_mbox/hive-user/201302.mbox/%3c8a0c145b-4db9-4d26-8613-8ca1bd741...@daum.net%3E. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5341) Link doesn't work. Needs to be updated as mentioned in the Description
[ https://issues.apache.org/jira/browse/HIVE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365113#comment-14365113 ] Fahd Khan commented on HIVE-5341: - https://code.google.com/p/hadoop-clusternet/source/browse/trunk/clusternet/thirdparty/data/ml-data.tar__0.gz?r=20 Link doesn't work. Needs to be updated as mentioned in the Description -- Key: HIVE-5341 URL: https://issues.apache.org/jira/browse/HIVE-5341 Project: Hive Issue Type: Bug Components: Documentation Reporter: Rakesh Chouhan Assignee: Lefty Leverenz Labels: documentation Go to.. Apache HIVE Getting Started Documentation https://cwiki.apache.org/confluence/display/Hive/GettingStarted Under Section ... Simple Example Use Cases MovieLens User Ratings wget http://www.grouplens.org/system/files/ml-data.tar+0.gz The link mentioned as per the document does not work. It needs to be updated to the below URL. http://www.grouplens.org/sites/www.grouplens.org/external_files/data/ml-data.tar.gz I am setting this defect's priority as a Blocker because, user's will not be able to continue their hands on exercises, unless they find the correct URL to download the mentioned file. Referenced from: http://mail-archives.apache.org/mod_mbox/hive-user/201302.mbox/%3c8a0c145b-4db9-4d26-8613-8ca1bd741...@daum.net%3E. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9697) Hive on Spark is not as aggressive as MR on map join [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365114#comment-14365114 ] Xuefu Zhang commented on HIVE-9697: --- It seems that we all agree that rawDataSize is more practical for Spark. Could anyone give a summary on if it's the default or how to make it as default? If code change is required, we can propose a patch here. Thanks. Hive on Spark is not as aggressive as MR on map join [Spark Branch] --- Key: HIVE-9697 URL: https://issues.apache.org/jira/browse/HIVE-9697 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xin Hao We have a finding during running some Big-Bench cases: when the same small table size threshold is used, Map Join operator will not be generated in Stage Plans for Hive on Spark, while will be generated for Hive on MR. For example, When we run BigBench Q25, the meta info of one input ORC table is as below: totalSize=1748955 (about 1.5M) rawDataSize=123050375 (about 120M) If we use the following parameter settings, set hive.auto.convert.join=true; set hive.mapjoin.smalltable.filesize=2500; set hive.auto.convert.join.noconditionaltask=true; set hive.auto.convert.join.noconditionaltask.size=1; (100M) Map Join will be enabled for Hive on MR mode, while will not be enabled for Hive on Spark. We found that for Hive on MR, the HDFS file size for the table (ContentSummary.getLength(), should approximate the value of ‘totalSize’) will be used to compare with the threshold 100M (smaller than 100M), while for Hive on Spark 'rawDataSize' will be used to compare with the threshold 100M (larger than 100M). That's why MapJoin is not enabled for Hive on Spark for this case. And as a result Hive on Spark will get much lower performance data than Hive on MR for this case. When we set hive.auto.convert.join.noconditionaltask.size=15000; (150M), MapJoin will be enabled for Hive on Spark mode, and Hive on Spark will have similar performance data with Hive on MR by then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7018) Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others
[ https://issues.apache.org/jira/browse/HIVE-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365141#comment-14365141 ] Aihua Xu commented on HIVE-7018: I think it's just removing the ending ',' , not removing the complete line. Table and Partition tables have column LINK_TARGET_ID in Mysql scripts but not others - Key: HIVE-7018 URL: https://issues.apache.org/jira/browse/HIVE-7018 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Yongzhi Chen Attachments: HIVE-7018.1.patch It appears that at least postgres and oracle do not have the LINK_TARGET_ID column while mysql does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9990) TestMultiSessionsHS2WithLocalClusterSpark is failing
[
https://issues.apache.org/jira/browse/HIVE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9990:
--
Description:
At least sometimes. I can reproduce it with mvn test
-Dtest=TestMultiSessionsHS2WithLocalClusterSpark -Phadoop-2 consistently on my
local box (both trunk and spark branch).
{code}
---
T E S T S
---
Running org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.438 sec
FAILURE! - in org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
testSparkQuery(org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark)
Time elapsed: 21.514 sec ERROR!
java.util.concurrent.ExecutionException: java.sql.SQLException: Error while
processing statement: FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.verifyResult(TestMultiSessionsHS2WithLocalClusterSpark.java:244)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testKvQuery(TestMultiSessionsHS2WithLocalClusterSpark.java:220)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.access$000(TestMultiSessionsHS2WithLocalClusterSpark.java:53)
{code}
The error was also seen in HIVE-9934 test run.
was:
At least sometimes. I can reproduce it with mvn test
-Dtest=TestMultiSessionsHS2WithLocalClusterSpark -Phadoop-2 consistently on my
local box.
{code}
---
T E S T S
---
Running org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.438 sec
FAILURE! - in org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
testSparkQuery(org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark)
Time elapsed: 21.514 sec ERROR!
java.util.concurrent.ExecutionException: java.sql.SQLException: Error while
processing statement: FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.verifyResult(TestMultiSessionsHS2WithLocalClusterSpark.java:244)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testKvQuery(TestMultiSessionsHS2WithLocalClusterSpark.java:220)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.access$000(TestMultiSessionsHS2WithLocalClusterSpark.java:53)
{code}
The error was also seen in HIVE-9934 test run.
TestMultiSessionsHS2WithLocalClusterSpark is failing
Key: HIVE-9990
URL: https://issues.apache.org/jira/browse/HIVE-9990
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: 1.2.0
Reporter: Xuefu Zhang
At least sometimes. I can reproduce it with mvn test
-Dtest=TestMultiSessionsHS2WithLocalClusterSpark -Phadoop-2 consistently on
my local box (both trunk and spark branch).
{code}
---
T E S T S
---
Running org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.438 sec
FAILURE! - in
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
testSparkQuery(org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark)
Time elapsed: 21.514 sec ERROR!
java.util.concurrent.ExecutionException: java.sql.SQLException: Error while
processing statement: FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.verifyResult(TestMultiSessionsHS2WithLocalClusterSpark.java:244)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testKvQuery(TestMultiSessionsHS2WithLocalClusterSpark.java:220)
at
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.access$000(TestMultiSessionsHS2WithLocalClusterSpark.java:53)
{code}
The error was also seen in HIVE-9934 test run.
--
This message was sent by
[jira] [Updated] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[ https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9934: -- Attachment: HIVE-9934.3.patch Attached the same patch for another test run. Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password -- Key: HIVE-9934 URL: https://issues.apache.org/jira/browse/HIVE-9934 Project: Hive Issue Type: Bug Components: Security Affects Versions: 1.1.0 Reporter: Chao Assignee: Chao Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch, HIVE-9934.3.patch Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password. See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html “If you supply an empty string, an empty byte/char array, or null to the Context.SECURITY_CREDENTIALS environment property, then the authentication mechanism will be none. This is because the LDAP requires the password to be nonempty for simple authentication. The protocol automatically converts the authentication to none if a password is not supplied.” Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a NamingException being thrown during creation of initial context, it does not fail when the context result is an “unauthenticated” positive response from the LDAP server. The end result is, one can authenticate with HiveServer2 using the LdapAuthenticationProviderImpl with only a user name and an empty password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9957) Hive 1.1.0 not compatible with Hadoop 2.4.0
[ https://issues.apache.org/jira/browse/HIVE-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365216#comment-14365216 ] Sergio Peña commented on HIVE-9957: --- Yes. Encryption was merged on 1.1.0. Hive 1.1.0 not compatible with Hadoop 2.4.0 --- Key: HIVE-9957 URL: https://issues.apache.org/jira/browse/HIVE-9957 Project: Hive Issue Type: Bug Components: Encryption Reporter: Vivek Shrivastava Assignee: Sergio Peña Labels: TODOC1.2 Fix For: 1.2.0 Attachments: HIVE-9957.1.patch Getting this exception while accessing data through Hive. Exception in thread main java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.DFSClient.getKeyProvider()Lorg/apache/hadoop/crypto/key/KeyProvider; at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.init(Hadoop23Shims.java:1152) at org.apache.hadoop.hive.shims.Hadoop23Shims.createHdfsEncryptionShim(Hadoop23Shims.java:1279) at org.apache.hadoop.hive.ql.session.SessionState.getHdfsEncryptionShim(SessionState.java:392) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1756) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1875) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1689) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1427) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10132) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10147) at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9945) FunctionTask.conf hides Task.conf field
[ https://issues.apache.org/jira/browse/HIVE-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365376#comment-14365376 ] Alan Gates commented on HIVE-9945: -- +1 FunctionTask.conf hides Task.conf field --- Key: HIVE-9945 URL: https://issues.apache.org/jira/browse/HIVE-9945 Project: Hive Issue Type: Bug Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Priority: Minor Attachments: HIVE-9945.1.patch Task class has protected field conf. FunctionTask can use it instead of creating another conf field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error
[
https://issues.apache.org/jira/browse/HIVE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-9991:
--
Attachment: HIVE-9991.1.patch
Cannot do a SELECT on external tables that are on S3 due to Encryption error
Key: HIVE-9991
URL: https://issues.apache.org/jira/browse/HIVE-9991
Project: Hive
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña
Attachments: HIVE-9991.1.patch
I cannot do any select query on external tables that are not part of HDFS.
For example S3.
{code}
Select * from my_table limit 10;
FAILED: SemanticException Unable to determine if s3n://my-bucket/is
encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/,
expected: hdfs://0.0.0.0:8020
{code}
This error is due to a internal function that checks if a table is encrypted
or not. This is only supported on HDFS files, but the check is happening on
any external table as well causing the above error.
To fix this, we should check for encrypted tables only for HDFS tables. And
skip the check for any other file schema.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9934) Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to none, allowing authentication without password
[
https://issues.apache.org/jira/browse/HIVE-9934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365402#comment-14365402
]
Hive QA commented on HIVE-9934:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705088/HIVE-9934.3.patch
{color:green}SUCCESS:{color} +1 7770 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3056/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3056/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3056/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705088 - PreCommit-HIVE-TRUNK-Build
Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to
degrade the authentication mechanism to none, allowing authentication
without password
--
Key: HIVE-9934
URL: https://issues.apache.org/jira/browse/HIVE-9934
Project: Hive
Issue Type: Bug
Components: Security
Affects Versions: 1.1.0
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9934.1.patch, HIVE-9934.2.patch, HIVE-9934.3.patch,
HIVE-9934.3.patch
Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to
degrade the authentication mechanism to none, allowing authentication
without password.
See: http://docs.oracle.com/javase/jndi/tutorial/ldap/security/simple.html
“If you supply an empty string, an empty byte/char array, or null to the
Context.SECURITY_CREDENTIALS environment property, then the authentication
mechanism will be none. This is because the LDAP requires the password to
be nonempty for simple authentication. The protocol automatically converts
the authentication to none if a password is not supplied.”
Since the LdapAuthenticationProviderImpl.Authenticate method is relying on a
NamingException being thrown during creation of initial context, it does not
fail when the context result is an “unauthenticated” positive response from
the LDAP server. The end result is, one can authenticate with HiveServer2
using the LdapAuthenticationProviderImpl with only a user name and an empty
password.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9555) assorted ORC refactorings for LLAP on trunk
[ https://issues.apache.org/jira/browse/HIVE-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366676#comment-14366676 ] Prasanth Jayachandran commented on HIVE-9555: - And for https://github.com/apache/hive/blob/llap/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/orc/stream/readers/StringStreamReader.java#L37 assorted ORC refactorings for LLAP on trunk --- Key: HIVE-9555 URL: https://issues.apache.org/jira/browse/HIVE-9555 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9555.01.patch, HIVE-9555.02.patch, HIVE-9555.03.patch, HIVE-9555.04.patch, HIVE-9555.05.patch, HIVE-9555.06.patch, HIVE-9555.07.patch, HIVE-9555.patch To minimize conflicts and given that ORC is being developed rapidly on trunk, I would like to refactor some parts of ORC in advance based on the changes in LLAP branch. Mostly it concerns making parts of ORC code (esp. SARG, but also some internal methods) more modular and easier to use from alternative codepaths. There's also significant change to how data reading is handled - BufferChunk inherits from DiskRange; the reader receives a list of DiskRange-s (as before), but instead of making a list of buffer chunks it replaces ranges with buffer chunks in the original (linked) list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9975) Renaming a nonexisting partition should not throw out NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366687#comment-14366687
]
Hive QA commented on HIVE-9975:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705245/HIVE-9975.2.patch
{color:green}SUCCESS:{color} +1 7770 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3067/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3067/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3067/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705245 - PreCommit-HIVE-TRUNK-Build
Renaming a nonexisting partition should not throw out NullPointerException
--
Key: HIVE-9975
URL: https://issues.apache.org/jira/browse/HIVE-9975
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
Priority: Minor
Attachments: HIVE-9975.1.patch, HIVE-9975.2.patch, HIVE-9975.patch
Renaming a nonexisting partition should not throw out NullPointerException.
create table testpart (col1 int, col2 string, col3 string) partitioned by
(part string);
alter table testpart partition (part = 'nonexisting') rename to partition
(part = 'existing');
we get NPE like following:
{code}
15/03/16 10:16:11 ERROR exec.DDLTask: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.DDLTask.renamePartition(DDLTask.java:944)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:350)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1642)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1402)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1187)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1053)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9555) assorted ORC refactorings for LLAP on trunk
[ https://issues.apache.org/jira/browse/HIVE-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366672#comment-14366672 ] Prasanth Jayachandran commented on HIVE-9555: - Its needed for https://github.com/apache/hive/blob/llap/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/orc/stream/readers/CharacterStreamReader.java#L37 to handle char and varchar in the same class. assorted ORC refactorings for LLAP on trunk --- Key: HIVE-9555 URL: https://issues.apache.org/jira/browse/HIVE-9555 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HIVE-9555.01.patch, HIVE-9555.02.patch, HIVE-9555.03.patch, HIVE-9555.04.patch, HIVE-9555.05.patch, HIVE-9555.06.patch, HIVE-9555.07.patch, HIVE-9555.patch To minimize conflicts and given that ORC is being developed rapidly on trunk, I would like to refactor some parts of ORC in advance based on the changes in LLAP branch. Mostly it concerns making parts of ORC code (esp. SARG, but also some internal methods) more modular and easier to use from alternative codepaths. There's also significant change to how data reading is handled - BufferChunk inherits from DiskRange; the reader receives a list of DiskRange-s (as before), but instead of making a list of buffer chunks it replaces ranges with buffer chunks in the original (linked) list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
