[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389189#comment-16389189
]
Hive QA commented on HIVE-18875:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913262/HIVE-18875.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 67 failed/errored test(s), 13482 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-18864:
Status: Patch Available (was: Open)
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389175#comment-16389175
]
Sankar Hariappan commented on HIVE-18864:
-
Attached 02.patch with
* Redundant code path removed.
* Fixed check style issues
Test failures in previous build are irrelevant to this patch.
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-18864:
Attachment: HIVE-18864.02.patch
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-18864:
Status: Open (was: Patch Available)
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Patch Available (was: Open) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: HIVE-18871.1.patch > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Open (was: Patch Available) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) >
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: (was: HIVE-18871.1.patch) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) >
[jira] [Commented] (HIVE-18891) I am trying to write a basic query in hive and it is giving me error
[ https://issues.apache.org/jira/browse/HIVE-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389134#comment-16389134 ] Amit Chauhan commented on HIVE-18891: - Also if we run same query with and condition it is running perfectly Explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect col2 FROM Table2) and col2 LIKE 'abc'; OK > I am trying to write a basic query in hive and it is giving me error > - > > Key: HIVE-18891 > URL: https://issues.apache.org/jira/browse/HIVE-18891 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Amit Chauhan >Priority: Trivial > > Query : > explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect col2 > FROM Table2) or col2 LIKE 'abc'; > FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery > Expression ''abc'': Only SubQuery expressions that are top level conjuncts > are allowed. > > if i am running query in standalone like the 2 ways mentioned below they are > running: > hive> explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect > col2 FROM Table2) > OK > hive> explain select * from Table1 WHERE col2 LIKE 'abc'; > OK > can you please help why the query with both conditions is not running. > > hiver version is : > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18891) I am trying to write a basic query in hive and it is giving me error
[ https://issues.apache.org/jira/browse/HIVE-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Chauhan updated HIVE-18891: Description: Query : explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect col2 FROM Table2) or col2 LIKE 'abc'; FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery Expression ''abc'': Only SubQuery expressions that are top level conjuncts are allowed. if i am running query in standalone like the 2 ways mentioned below they are running: hive> explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect col2 FROM Table2) OK hive> explain select * from Table1 WHERE col2 LIKE 'abc'; OK can you please help why the query with both conditions is not running. hiver version is : hive> !hive --version; Hive 1.2.1000.2.5.6.0-40 was: Query : explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM UDM_PRD_ITVM.SALES_PERSON) or SLS_PRSN_ID_NEW LIKE 'abc'; FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery Expression ''abc'': Only SubQuery expressions that are top level conjuncts are allowed. if i am running query in standalone like the 2 ways mentioned below they are running: hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM UDM_PRD_ITVM.SALES_PERSON) ; OK hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE a.SLS_PRSN_ID_NEW LIKE 'abc' ; OK can you please help why the query with both conditions is not running. hiver version is : hive> !hive --version; Hive 1.2.1000.2.5.6.0-40 > I am trying to write a basic query in hive and it is giving me error > - > > Key: HIVE-18891 > URL: https://issues.apache.org/jira/browse/HIVE-18891 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Amit Chauhan >Priority: Trivial > > Query : > explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect col2 > FROM Table2) or col2 LIKE 'abc'; > FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery > Expression ''abc'': Only SubQuery expressions that are top level conjuncts > are allowed. > > if i am running query in standalone like the 2 ways mentioned below they are > running: > hive> explain select * from Table1 WHERE COALESCE(Col1, 'abc') in ( SELect > col2 FROM Table2) > OK > hive> explain select * from Table1 WHERE col2 LIKE 'abc'; > OK > can you please help why the query with both conditions is not running. > > hiver version is : > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-18891) I am trying to write a basic query in hive and it is giving me error
[ https://issues.apache.org/jira/browse/HIVE-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Chauhan updated HIVE-18891: Comment: was deleted (was: Also when i replace or with and condition they query runs: hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE a.SLS_PRSN_ID_NEW LIKE '1234' and COALESCE(a.SLS_PRSN_ID_NEW,'3344') in ( select SLS_PRSN_ID as t FROM UDM_PRD_ITVM.SALES_PERSON); OK) > I am trying to write a basic query in hive and it is giving me error > - > > Key: HIVE-18891 > URL: https://issues.apache.org/jira/browse/HIVE-18891 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Amit Chauhan >Priority: Trivial > > Query : > explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE > COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM > UDM_PRD_ITVM.SALES_PERSON) or SLS_PRSN_ID_NEW LIKE 'abc'; > FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery > Expression ''abc'': Only SubQuery expressions that are top level conjuncts > are allowed. > > if i am running query in standalone like the 2 ways mentioned below they are > running: > hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE > COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM > UDM_PRD_ITVM.SALES_PERSON) ; > OK > hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE > a.SLS_PRSN_ID_NEW LIKE 'abc' ; > OK > can you please help why the query with both conditions is not running. > > hiver version is : > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: HIVE-18871.1.patch > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: (was: HIVE-18871.1.patch) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events
[ https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18781: --- Attachment: HIVE-18781.03.patch > Create/Replicate Open, Commit (without writes) and Abort Txn events > --- > > Key: HIVE-18781 > URL: https://issues.apache.org/jira/browse/HIVE-18781 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch, > HIVE-18781.03.patch > > > *EVENT_OPEN_TXN:* > *Source Warehouse:* > - Create new event type EVENT_OPEN_TXN with related message format etc. > - When any transaction is opened either by auto-commit mode or > multi-statement mode, need to capture this event. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Open a txn in target warehouse. > - Create a map of source txn ID against target txn ID and persist the same > in metastore. There should be one map per replication policy (DBName.* incase > of DB level replication, DBName.TableName incase of table level replication) > > *EVENT_COMMIT_TXN (Without writes)* > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > - Create EVENT_COMMIT_TXN event type with corresponding message format etc. > *Target warehouse:* > - Repl load should read this event from the dump. > - Validate the source txn ID from the event using the Source-Target Txn ID > map maintained in target metastore. Also, need to check if corresponding > target txn ID is valid. > - If valid, then apply the event and commit the corresponding target > transaction. > - This new event should be idempotent such that if it is applied twice, then > second time it should be loop. > > *EVENT_ABORT_TXN* > Source Warehouse: > - Create new event type EVENT_ABORT_TXN with related message format etc. > - Capture this event when abort the txn. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Validate if source txn ID from the event is there in the source-target txn > ID map. If not there, just noop the event. > - If valid, then Abort the corresponding target txn and remove the entry > from source-target txn ID map. > All these new events should be idempotent such that if it is applied twice, > then second time it should be noop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez
[
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389119#comment-16389119
]
Hive QA commented on HIVE-18875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9525/dev-support/hive-personality.sh
|
| git revision | master / cd8eda8 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9525/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9525/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable SMB Join by default in Tez
> -
>
> Key: HIVE-18875
> URL: https://issues.apache.org/jira/browse/HIVE-18875
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18875.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389117#comment-16389117 ] Vaibhav Gumashta commented on HIVE-18264: - [~akolb] I've converted all tasks completed so far and upcoming ones as subtasks here: HIVE-16520. The jira already has a link to the initially proposed design (which we are improving on as we work through different issues) which I have updated. I'll shortly post an updated patch incorporating rb feedback. > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389113#comment-16389113 ] Peter Vary commented on HIVE-18883: --- Findbugs binary has to be installed on the ptest server and $FINDBUGS_HOME has to be set before starting yetus. Check out this page: [https://cwiki.apache.org/confluence/display/Hive/Running+Yetus] Thanks for taking on this. It is on my TODO list since introducing yetus, but I never seem to find time to do this. If you need any help, just ask. Thanks, Peter > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Patch Available (was: Open) It's better to check the scheme of the src and chose the right file system in function DagUtils.localizeResource > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Commented] (HIVE-18891) I am trying to write a basic query in hive and it is giving me error
[ https://issues.apache.org/jira/browse/HIVE-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389099#comment-16389099 ] Amit Chauhan commented on HIVE-18891: - Also when i replace or with and condition they query runs: hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE a.SLS_PRSN_ID_NEW LIKE '1234' and COALESCE(a.SLS_PRSN_ID_NEW,'3344') in ( select SLS_PRSN_ID as t FROM UDM_PRD_ITVM.SALES_PERSON); OK > I am trying to write a basic query in hive and it is giving me error > - > > Key: HIVE-18891 > URL: https://issues.apache.org/jira/browse/HIVE-18891 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Amit Chauhan >Priority: Trivial > > Query : > explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE > COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM > UDM_PRD_ITVM.SALES_PERSON) or SLS_PRSN_ID_NEW LIKE 'abc'; > FAILED: SemanticException [Error 10249]: Line 1:97 Unsupported SubQuery > Expression ''abc'': Only SubQuery expressions that are top level conjuncts > are allowed. > > if i am running query in standalone like the 2 ways mentioned below they are > running: > hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS WHERE > COALESCE(SLS_PRSN_ID_NEW, 'abc') in ( SELect SLS_PRSN_ID FROM > UDM_PRD_ITVM.SALES_PERSON) ; > OK > hive> explain select * from CCRS_PRDSTG_TBLS.CCRS_SALESREP_ACTS a WHERE > a.SLS_PRSN_ID_NEW LIKE 'abc' ; > OK > can you please help why the query with both conditions is not running. > > hiver version is : > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18436) Upgrade to Spark 2.3.0
[
https://issues.apache.org/jira/browse/HIVE-18436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389097#comment-16389097
]
Hive QA commented on HIVE-18436:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913256/HIVE-18436.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 102 failed/errored test(s), 13878 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_collect_set_unsupported]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_array_contains_wrong2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=95)
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: HIVE-18871.1.patch > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Attachment: (was: HIVE-18871-001.patch) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871.1.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-18871: -- Status: Open (was: Patch Available) > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871-001.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) > [hive-cli-2.1.1.jar:2.1.1] > at
[jira] [Updated] (HIVE-18610) Performance: ListKeyWrapper does not check for hashcode equals, before comparing members
[
https://issues.apache.org/jira/browse/HIVE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18610:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
> Performance: ListKeyWrapper does not check for hashcode equals, before
> comparing members
>
>
> Key: HIVE-18610
> URL: https://issues.apache.org/jira/browse/HIVE-18610
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Labels: complextype, performance
> Fix For: 3.0.0
>
> Attachments: HIVE-18610.1.patch, HIVE-18610.2.patch
>
>
> ListKeyWrapper::equals()
> {code}
> @Override
> public boolean equals(Object obj) {
> if (!(obj instanceof ListKeyWrapper)) {
> return false;
> }
> Object[] copied_in_hashmap = ((ListKeyWrapper) obj).keys;
> return equalComparer.areEqual(copied_in_hashmap, keys);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18888) Replace synchronizedMap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-1: -- Status: Patch Available (was: Open) > Replace synchronizedMap with ConcurrentHashMap > -- > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.3.3 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-1.01.patch > > > There are a bunch of places that use Collections.synchronizedMap instead of > ConcurrentHashMap which are better. We should search/replace the uses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18888) Replace synchronizedMap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-1: -- Attachment: HIVE-1.01.patch > Replace synchronizedMap with ConcurrentHashMap > -- > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.3.3 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-1.01.patch > > > There are a bunch of places that use Collections.synchronizedMap instead of > ConcurrentHashMap which are better. We should search/replace the uses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18436) Upgrade to Spark 2.3.0
[
https://issues.apache.org/jira/browse/HIVE-18436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389062#comment-16389062
]
Hive QA commented on HIVE-18436:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9523/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9523/yetus/patch-asflicense-problems.txt
|
| modules | C: . itests ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9523/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Spark 2.3.0
> --
>
> Key: HIVE-18436
> URL: https://issues.apache.org/jira/browse/HIVE-18436
> Project: Hive
> Issue Type: Task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18436.1.patch, HIVE-18436.2.patch,
> HIVE-18436.3.patch
>
>
> Branching has been completed. Release candidates should be published soon.
> Might be a while before the actual release, but at least we get to identify
> any issues early.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18849) Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-18849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389038#comment-16389038 ] Мирон commented on HIVE-18849: -- This works with java 9 deployed as following: C:\dev\java standing on this directory C:\dev\git\hadoop\hive\standalone-metastore\target\classes with package.jdo under same java -classpath C:\dev\git\hadoop\hive\standalone-metastore\target\classes;C:\Users\hadoop\.m2\repository\org\datanucleus\datanucleus-core\5.1.7\datanucleus-core-5.1.7.jar;C:\Users\hadoop\.m2\repository\org\datanucleus\datanucleus-rdbms\5.1.6\datanucleus-rdbms-5.1.6.jar;C:\Users\hadoop\.m2\repository\org\datanucleus\javax.jdo\3.2.0-m8\javax.jdo-3.2.0-m8.jar;C:\Users\hadoop\.m2\repository\org\datanucleus\datanucleus-api-jdo\5.1.4\datanucleus-api-jdo-5.1.4.jar org.datanucleus.enhancer.DataNucleusEnhancer -v -api JDO C:\dev\git\hadoop\hive\standalone-metastore\target\classes\org\apache\hadoop\hive\metastore\model\MDatabase.class DataNucleusEnhancer version 5.1.6 ( latest release in the repository ) invoked manually works without issues, too. In other words, as far as code / tool goes, enhancer works. Takes a bit of an effort to learn the infrastructure to get to the details. Feel free to take over at any time, since I can be pulled into something else at any moment. For the moment assigning to self. > Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: > java.lang.NullPointerException > - > > Key: HIVE-18849 > URL: https://issues.apache.org/jira/browse/HIVE-18849 > Project: Hive > Issue Type: Bug >Reporter: Мирон >Assignee: Мирон >Priority: Blocker > Attachments: Issue_Build_Complete_Log.txt, Issue_Stack_Trace.txt > > > Please see attached stack trace, both, brief and complete capture, both using > -X verbose output maven flag. > Irrespective of the true cause, it would be very nice if this message of > build tool ( maven ) > -- > Caused by: java.lang.NullPointerException > at com.sun.tools.javac.main.JavaCompiler.readSourceFile > (JavaCompiler.java:825) > at > com.sun.tools.javac.processing.JavacProcessingEnvironment$ImplicitCompleter.complete > (JavacProcessingEnvironment.java:1510) > at com.sun.tools.javac.code.Symbol.complete (Symbol.java:633) > -- > carried actual file name, that was attempted reading and failed with > exception. > > Git repository cloned from [https://github.com/apache/hive.git] yesterday - > today overnight. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18433) Upgrade version of com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/HIVE-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389036#comment-16389036
]
Hive QA commented on HIVE-18433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913257/HIVE-18433.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9522/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9522/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9522/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-07 05:01:40.554
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9522/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-07 05:01:40.557
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at abede8e Revert "HIVE-18789: Disallow embedded element in
UDFXPathUtil (Daniel Dai, reviewed by Thejas Nair)"
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at abede8e Revert "HIVE-18789: Disallow embedded element in
UDFXPathUtil (Daniel Dai, reviewed by Thejas Nair)"
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-07 05:01:42.160
+ rm -rf ../yetus_PreCommit-HIVE-Build-9522
+ mkdir ../yetus_PreCommit-HIVE-Build-9522
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9522
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9522/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-hcatalog: Could not resolve
dependencies for project
org.apache.hive.hcatalog:hive-hcatalog:pom:3.0.0-SNAPSHOT: The following
artifacts could not be resolved:
com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.9.2,
com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.9.2,
com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.9.2: Could not find
artifact com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.9.2
in datanucleus (http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-hcatalog
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12913257 - PreCommit-HIVE-Build
> Upgrade version of com.fasterxml.jackson
>
>
> Key: HIVE-18433
> URL: https://issues.apache.org/jira/browse/HIVE-18433
> Project: Hive
> Issue Type: Task
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18433.1.patch, HIVE-18433.2.patch,
> HIVE-18433.3.patch, HIVE-18433.4.patch
>
>
> Let's upgrade to version 2.9.2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18872) Projection is not pushed properly when query involves multiple tables
[
https://issues.apache.org/jira/browse/HIVE-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389030#comment-16389030
]
Hive QA commented on HIVE-18872:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913234/HIVE-18872.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 100 failed/errored test(s), 13482 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Assigned] (HIVE-18849) Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-18849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Мирон reassigned HIVE-18849: Assignee: Мирон > Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: > java.lang.NullPointerException > - > > Key: HIVE-18849 > URL: https://issues.apache.org/jira/browse/HIVE-18849 > Project: Hive > Issue Type: Bug >Reporter: Мирон >Assignee: Мирон >Priority: Blocker > Attachments: Issue_Build_Complete_Log.txt, Issue_Stack_Trace.txt > > > Please see attached stack trace, both, brief and complete capture, both using > -X verbose output maven flag. > Irrespective of the true cause, it would be very nice if this message of > build tool ( maven ) > -- > Caused by: java.lang.NullPointerException > at com.sun.tools.javac.main.JavaCompiler.readSourceFile > (JavaCompiler.java:825) > at > com.sun.tools.javac.processing.JavacProcessingEnvironment$ImplicitCompleter.complete > (JavacProcessingEnvironment.java:1510) > at com.sun.tools.javac.code.Symbol.complete (Symbol.java:633) > -- > carried actual file name, that was attempted reading and failed with > exception. > > Git repository cloned from [https://github.com/apache/hive.git] yesterday - > today overnight. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18849) Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-18849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18849 started by Мирон. > Java 9 Building "mvn clean package -Pdist -rf :hive-common" get Caused by: > java.lang.NullPointerException > - > > Key: HIVE-18849 > URL: https://issues.apache.org/jira/browse/HIVE-18849 > Project: Hive > Issue Type: Bug >Reporter: Мирон >Assignee: Мирон >Priority: Blocker > Attachments: Issue_Build_Complete_Log.txt, Issue_Stack_Trace.txt > > > Please see attached stack trace, both, brief and complete capture, both using > -X verbose output maven flag. > Irrespective of the true cause, it would be very nice if this message of > build tool ( maven ) > -- > Caused by: java.lang.NullPointerException > at com.sun.tools.javac.main.JavaCompiler.readSourceFile > (JavaCompiler.java:825) > at > com.sun.tools.javac.processing.JavacProcessingEnvironment$ImplicitCompleter.complete > (JavacProcessingEnvironment.java:1510) > at com.sun.tools.javac.code.Symbol.complete (Symbol.java:633) > -- > carried actual file name, that was attempted reading and failed with > exception. > > Git repository cloned from [https://github.com/apache/hive.git] yesterday - > today overnight. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18889) update all parts of Hive to use the same Guava version
[ https://issues.apache.org/jira/browse/HIVE-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388984#comment-16388984 ] Sergey Shelukhin commented on HIVE-18889: - Hm... or fix the tests? > update all parts of Hive to use the same Guava version > -- > > Key: HIVE-18889 > URL: https://issues.apache.org/jira/browse/HIVE-18889 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18889.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18872) Projection is not pushed properly when query involves multiple tables
[
https://issues.apache.org/jira/browse/HIVE-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388982#comment-16388982
]
Hive QA commented on HIVE-18872:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 13 unchanged - 4
fixed = 13 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9521/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9521/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9521/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Projection is not pushed properly when query involves multiple tables
> -
>
> Key: HIVE-18872
> URL: https://issues.apache.org/jira/browse/HIVE-18872
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18872.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18889) update all parts of Hive to use the same Guava version
[ https://issues.apache.org/jira/browse/HIVE-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388981#comment-16388981 ] Ashutosh Chauhan commented on HIVE-18889: - Druid unit tests failed with 19. Thats why its at 16. So, if 19 doesn't work, we need to move to 16 through out. [~bslim] correct? > update all parts of Hive to use the same Guava version > -- > > Key: HIVE-18889 > URL: https://issues.apache.org/jira/browse/HIVE-18889 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18889.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x
[
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388972#comment-16388972
]
Ashutosh Chauhan commented on HIVE-18861:
-
Patch needs to be named according to:
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing
> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating
> classpath problems on hadoop 3.x
>
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP,
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18835) JDBC standalone jar download link in ambari
[ https://issues.apache.org/jira/browse/HIVE-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388968#comment-16388968 ] Ashutosh Chauhan commented on HIVE-18835: - Don't throw IllegalStateException, that will bring down HS2. Either return error to user or log error message. > JDBC standalone jar download link in ambari > --- > > Key: HIVE-18835 > URL: https://issues.apache.org/jira/browse/HIVE-18835 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-18835.patch > > > Let HS2 offer the file for download, so that Ambari can create link on it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18060) UpdateInputAccessTimeHook fails for non-current database
[
https://issues.apache.org/jira/browse/HIVE-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388960#comment-16388960
]
Hive QA commented on HIVE-18060:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913239/HIVE-18060.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13087 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-14183) hive 2.1.0 beeline cannot show verbose log
[ https://issues.apache.org/jira/browse/HIVE-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388952#comment-16388952 ] Mike Li commented on HIVE-14183: any update? I need this useful feature. > hive 2.1.0 beeline cannot show verbose log > -- > > Key: HIVE-14183 > URL: https://issues.apache.org/jira/browse/HIVE-14183 > Project: Hive > Issue Type: Bug > Components: Beeline, HiveServer2 >Affects Versions: 2.1.0 >Reporter: jack >Assignee: Prasanth Jayachandran >Priority: Major > > In Hive 1.2.1 , config the log verbose , the beeline client will show the > verbose log like this: > 16/07/07 13:29:33 INFO mapreduce.Job: The url to track the job: > http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Starting Job = job_1467708727273_0035, > Tracking URL = http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Kill Command = > /hadoop/hadoop-2.7.2/bin/hadoop job -kill job_1467708727273_0035 > 16/07/07 13:30:07 INFO exec.Task: Hadoop job information for Stage-1: number > of mappers: 1; number of reducers: 1 > 16/07/07 13:30:07 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 16/07/07 13:30:07 INFO exec.Task: 2016-07-07 13:30:07,905 Stage-1 map = 0%, > reduce = 0% > 16/07/07 13:30:17 INFO exec.Task: 2016-07-07 13:30:17,757 Stage-1 map = 100%, > reduce = 0%, Cumulative CPU 2.39 sec > 16/07/07 13:30:28 > the verbose config: > > hive.server2.logging.operation.enabled > true > When true, HS2 will save operation logs and make them > available for clients > > > hive.server2.logging.operation.log.location > /hadooplog/apache-hive-1.2.1-bin/operation_logs > Top level directory where operation logs are stored if > logging functionality is enabled > > > hive.server2.logging.operation.level > VERBOSE > > Expects one of [none, execution, performance, verbose]. > HS2 operation logging mode available to clients to be set at session > level. > For this to work, hive.server2.logging.operation.enabled should be set > to true. > NONE: Ignore any logging > EXECUTION: Log completion of tasks > PERFORMANCE: Execution + Performance logs > VERBOSE: All logs > > > However, when upgrade to hive 2.1.0, with the same verbose configure, the > verbose log cannot show. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18890) Lower Logging for "Table not found" Error
[
https://issues.apache.org/jira/browse/HIVE-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388948#comment-16388948
]
Mike Drob commented on HIVE-18890:
--
Ha, you've been listening to my rants! Yes, catch-log-and-throw is generally an
anti-pattern, for those unfamiliar there's a short blog post at
https://rolf-engelhard.de/2013/04/logging-anti-patterns-part-ii/ that covers
this and others.
Looking at this particular code snippet, the log should likely be outside of
the if, before the {{return null}}, and the thrown HiveException at the end
doesn't need to repeat the exception's message.
> Lower Logging for "Table not found" Error
> -
>
> Key: HIVE-18890
> URL: https://issues.apache.org/jira/browse/HIVE-18890
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Minor
>
> https://github.com/apache/hive/blob/7cb31c03052b815665b3231f2e513b9e65d3ff8c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1105
> {code:java}
> // Get the table from metastore
> org.apache.hadoop.hive.metastore.api.Table tTable = null;
> try {
> tTable = getMSC().getTable(dbName, tableName);
> } catch (NoSuchObjectException e) {
> if (throwException) {
> LOG.error("Table " + tableName + " not found: " + e.getMessage());
> throw new InvalidTableException(tableName);
> }
> return null;
> } catch (Exception e) {
> throw new HiveException("Unable to fetch table " + tableName + ". " +
> e.getMessage(), e);
> }
> {code}
> We should throw an exception or log it, but not both. Right [~mdrob] ? ;)
> And in this case, we are generating scary ERROR level logging in the
> HiveServer2 logs needlessly. This should not be reported as an application
> error. It is a simple user error, indicated by catching the
> _NoSuchObjectException_ Throwable, that can always be ignored by the service.
> It is most likely a simple user typo of the table name. However, the more
> serious general _Exception_ is not logged. This is backwards.
> Please remove the _error_ level logging for the user error... or lower it to
> _debug_ level logging.
> Please include an _error_ level logging to the general Exception case, unless
> this Exception is being captured up the stack, somewhere else, and is being
> logged there at ERROR level logging.
> {code}
> -- Sample log messages found in HS2 logs
> 2018-03-02 10:26:40,363 ERROR hive.ql.metadata.Hive:
> [HiveServer2-Handler-Pool: Thread-4467]: Table default not found:
> default.default table not found
> 2018-03-02 10:26:40,367 ERROR hive.ql.metadata.Hive:
> [HiveServer2-Handler-Pool: Thread-4467]: Table default not found:
> default.default table not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388946#comment-16388946
]
Vihang Karajgaonkar commented on HIVE-17580:
Thanks [~owen.omalley] for explaining. In that case I think it is better to
break compatibility on TypeInfo.getCategory() method signature. There hasn't
been any released version of Hive which has TypeInfo annotated as public API so
this is as good a time to fix this as any. IMHO Category enum should not be in
ObjectInspector in the first place. Instead of bending backwards to accommodate
a design choice which doesn't make sense I think we should fix it. I propose we
should introduce a new enum called {{MetastoreTypeCategory}} in metastore and
change the signature of {{getCategory()}} method in {{TypeInfo}} to return
{{MetastoreTypeCategory}} instead of {{ObjectInspector.Category}}.
{{ObjectInspector.Category}} would be modified so that there are helper methods
to map one to another. The values of the enums will also remain the same, so as
long as someone is using this enum for switch statements it would just work
without any changes. However, if some project outside hive is using it for
equality checks, it will not work. Eg:
{{typeInfo.getCategory() == ObjectInspector.Category.PRIMITIVE}} will not work.
Instead users should use the utility methods to convert them and compare. So in
this case they could do a {{typeInfo.getCategory() ==
ObjectInspector.Category.PRIMITIVE.toMetastoreTypeCategory()}} to get the
previous behavior.
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch,
> HIVE-17580.05-standalone-metastore.patch,
> HIVE-17580.06-standalone-metastore.patch,
> HIVE-17580.07-standalone-metastore.patch,
> HIVE-17580.08-standalone-metastore.patch,
> HIVE-17580.09-standalone-metastore.patch,
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18571) stats issues for MM tables; ACID doesn't check state for CTAS
[ https://issues.apache.org/jira/browse/HIVE-18571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388930#comment-16388930 ] Eugene Koifman commented on HIVE-18571: --- +1 pending tests > stats issues for MM tables; ACID doesn't check state for CTAS > - > > Key: HIVE-18571 > URL: https://issues.apache.org/jira/browse/HIVE-18571 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18571.01.patch, HIVE-18571.02.patch, > HIVE-18571.03.patch, HIVE-18571.04.patch, HIVE-18571.05.patch, > HIVE-18571.06.patch, HIVE-18571.patch > > > There are multiple stats aggregation issues with MM tables. > Some simple stats are double counted and some stats (simple stats) are > invalid for ACID table dirs altogether. > I have a patch almost ready, need to fix some more stuff and clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388925#comment-16388925
]
Eugene Koifman commented on HIVE-18825:
---
I'm strongly against this.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18060) UpdateInputAccessTimeHook fails for non-current database
[
https://issues.apache.org/jira/browse/HIVE-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388921#comment-16388921
]
Hive QA commented on HIVE-18060:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9520/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9520/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9520/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> UpdateInputAccessTimeHook fails for non-current database
>
>
> Key: HIVE-18060
> URL: https://issues.apache.org/jira/browse/HIVE-18060
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18060.1.patch, HIVE-18060.2.patch
>
>
> Steps to reproduce:
> *STEP 1. Create DBs and tables*
> {code}
> hive> create database temp;
> hive> use temp;
> hive> create table test(id int);
> hive> create database temp2;
> hive> use temp2;
> hive> create table test2(id int);
> {code}
> *STEP 2. Set {{hive.exec.pre.hooks}}*
> {code}
> hive> set
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
> {code}
> *STEP 3. Use {{desc}}*
> {code}
> hive> use temp;
> hive> desc temp2.test2;
> {code}
> *EXPECTED RESULT*
> Code works fine and shows table info
> *ACTUAL RESULT*
> {code}
> FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found
> test2)
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> test2
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1258)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1209)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1196)
>
[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr
[
https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388915#comment-16388915
]
Rui Li commented on HIVE-15104:
---
[~stakiar], thanks for trying this out.
bq. The HiveKryoRegistrator still seems to be serializing the hashCode so where
are the actual savings coming from?
I didn't look deeply into kryo, but I think the reason is generic kryo SerDe
has some overhead to store class meta info, while
in {{HiveKryoRegistrator}} we just store the data. My earlier
[comment|https://issues.apache.org/jira/browse/HIVE-15104?focusedCommentId=16007788=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16007788]
shows custom SerDe can bring improvements for BytesWritable too.
bq. I'm not sure I understand why the performance should improve when
hive.spark.use.groupby.shuffle is set to false.
I guess the difference is due to the different shuffle we used -- if
{{hive.spark.use.groupby.shuffle}} is false, group-by-key shuffle is replaced
with repartition-and-sort-within-partition shuffle. And yes, the registrator is
same for the two cases.
bq. why do we need the hashCode after deserializing the data?
For MR, the hash code is not needed for deserialized HiveKey (see
HiveKey::hashCode), because when HiveKey is deserialized, it's already been
distributed to the proper reducer. For Spark, RDDs may get cached during the
execution. So if we deserialize a cached RDD and try to partition it to a
downstream reducer, we'll need the hash code available after deserialization.
> Hive on Spark generate more shuffle data than hive on mr
>
>
> Key: HIVE-15104
> URL: https://issues.apache.org/jira/browse/HIVE-15104
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Rui Li
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15104.1.patch, HIVE-15104.10.patch,
> HIVE-15104.2.patch, HIVE-15104.3.patch, HIVE-15104.4.patch,
> HIVE-15104.5.patch, HIVE-15104.6.patch, HIVE-15104.7.patch,
> HIVE-15104.8.patch, HIVE-15104.9.patch, TPC-H 100G.xlsx
>
>
> the same sql, running on spark and mr engine, will generate different size
> of shuffle data.
> i think it is because of hive on mr just serialize part of HiveKey, but hive
> on spark which using kryo will serialize full of Hivekey object.
> what is your opionion?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18889) update all parts of Hive to use the same Guava version
[ https://issues.apache.org/jira/browse/HIVE-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18889: Status: Patch Available (was: Open) [~bslim] [~ashutoshc] can you take a look? thnx > update all parts of Hive to use the same Guava version > -- > > Key: HIVE-18889 > URL: https://issues.apache.org/jira/browse/HIVE-18889 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18889.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18889) update all parts of Hive to use the same Guava version
[ https://issues.apache.org/jira/browse/HIVE-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18889: --- > update all parts of Hive to use the same Guava version > -- > > Key: HIVE-18889 > URL: https://issues.apache.org/jira/browse/HIVE-18889 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18889.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18889) update all parts of Hive to use the same Guava version
[ https://issues.apache.org/jira/browse/HIVE-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18889: Attachment: HIVE-18889.patch > update all parts of Hive to use the same Guava version > -- > > Key: HIVE-18889 > URL: https://issues.apache.org/jira/browse/HIVE-18889 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18889.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18716) Delete unnecessary parameters from TaskFactory
[
https://issues.apache.org/jira/browse/HIVE-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388910#comment-16388910
]
Hive QA commented on HIVE-18716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913235/HIVE-18716.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13064 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18831: Status: Patch Available (was: Open) > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-18831: --- Assignee: Sahil Takiar > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1631#comment-1631
]
Sahil Takiar commented on HIVE-18831:
-
Before this patch the console output would look like:
{code}
Job failed with org.apache.hadoop.hive.ql.metadata.HiveException: [Error
20003]: An error occurred when trying to close the Operator running your custom
script.
FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during
runtime. Please check stacktrace for the root cause.
{code}
Now it looks like:
{code}
FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to Spark
task failures: Job failed with
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error
occurred when trying to close the Operator running your custom script.
{code}
So pretty much just combined these two lines and cleaned up the error message a
bit.
> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18831: Attachment: HIVE-18831.1.patch > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18871) hive on tez execution error due to set hive.aux.jars.path to hdfs://
[ https://issues.apache.org/jira/browse/HIVE-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388865#comment-16388865 ] muxin commented on HIVE-18871: -- thanks,I too have met this issue on hive 1.2.1 > hive on tez execution error due to set hive.aux.jars.path to hdfs:// > > > Key: HIVE-18871 > URL: https://issues.apache.org/jira/browse/HIVE-18871 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.2.1 > Environment: hadoop 2.6.5 > hive 2.2.1 > tez 0.8.4 >Reporter: zhuwei >Assignee: zhuwei >Priority: Major > Attachments: HIVE-18871-001.patch > > > When set the properties > hive.aux.jars.path=hdfs://mycluster/apps/hive/lib/guava.jar > and hive.execution.engine=tez; execute any query will fail with below error > log: > exec.Task: Failed to execute tez graph. > java.lang.IllegalArgumentException: Wrong FS: > hdfs://mycluster/apps/hive/lib/guava.jar, expected: file:/// > at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:529) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > ~[hadoop-common-2.6.0.jar:?] > at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1905) > ~[hadoop-common-2.6.0.jar:?] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1007) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:252) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager$TezSessionPoolSession.openInternal(TezSessionPoolManager.java:622) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206) > ~[hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:283) > ~[hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:155) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) > [hive-exec-2.1.1.jar:2.1.1] > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) > [hive-exec-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:429) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:445) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) > [hive-cli-2.1.1.jar:2.1.1] > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) >
[jira] [Commented] (HIVE-18716) Delete unnecessary parameters from TaskFactory
[
https://issues.apache.org/jira/browse/HIVE-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388848#comment-16388848
]
Hive QA commented on HIVE-18716:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 7 new + 2004 unchanged - 6
fixed = 2011 total (was 2010) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9519/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9519/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9519/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9519/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Delete unnecessary parameters from TaskFactory
> --
>
> Key: HIVE-18716
> URL: https://issues.apache.org/jira/browse/HIVE-18716
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Gergely Hajós
>Assignee: Gergely Hajós
>Priority: Trivial
> Attachments: HIVE-18716.1.patch, HIVE-18716.2.patch,
> HIVE-18716.3.patch, HIVE-18716.4.patch, HIVE-18716.5.patch
>
>
> * In _TaskFactory class conf_ parameter is not used here
> {code:java}
> public static Task get(Class workClass,
> HiveConf conf)
> {code}
> * _get_ functions never called with vararg parameter _Task Serializable>... tasklist_
> * the following version of _get_ is always called with _setConf_ true from
> outside of _TaskFactory_ class
> {code:java}
> public static Task get(T work, HiveConf conf,
> boolean setConf, Task... tasklist)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388834#comment-16388834
]
Jesus Camacho Rodriguez commented on HIVE-18825:
I have been checking the code, and indeed changing the position for acquiring
the locks requires a lot of work. Although we extract the read entities while
parsing (and I guess we could extend that logic and extract write entities
too), the logic to extract locks, etc., is tightly linked to the query plan
object.
One question I have is: does the proposed change break any current existing use
case or do we need it only in case we want to support lock based concurrency
control?
If it does not break the current system implementation, I suggest we commit
this fix to unlock MV incremental rebuild, etc., and we create a follow-up JIRA
to move the logic to acquire locks to be executed after parsing and before rest
of compilation.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18888) Replace synchronizedMap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-1: - > Replace synchronizedMap with ConcurrentHashMap > -- > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.3.3 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > There are a bunch of places that use Collections.synchronizedMap instead of > ConcurrentHashMap which are better. We should search/replace the uses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18884:
-
Assignee: Andrew Sherman
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Sserver (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388830#comment-16388830
]
Hive QA commented on HIVE-18864:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913236/HIVE-18864.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 74 failed/errored test(s), 13879 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_format_number_wrong3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_format_number_wrong5]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_greatest_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_if_not_bool]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_if_wrong_args_len]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_in]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_instr_wrong_args_len]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_invalid]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_likeall_wrong1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_likeany_wrong1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_map_keys_arg_num]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_map_values_arg_type]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_max]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_next_day_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_next_day_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_nonexistent_resource]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_printf_wrong4]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_by_wrong1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong3]
(batchId=95)
[jira] [Assigned] (HIVE-18887) Improve preserving column stats for alter table commands
[ https://issues.apache.org/jira/browse/HIVE-18887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu reassigned HIVE-18887: --- > Improve preserving column stats for alter table commands > > > Key: HIVE-18887 > URL: https://issues.apache.org/jira/browse/HIVE-18887 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > > We are trying to preserve column stats for certain alter table commands, > while seems that current generic approach which compare the old columns > against the new columns and update for all the columns may not be efficient . > e.g., if we just rename the table, we should be able to update the name > itself. COL_STATS table somehow contains DB_Name and Table_Name. If those > tables don't have these columns, certain commands don't even need to update > these tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388769#comment-16388769
]
Gopal V commented on HIVE-18886:
This is the fall-back exception
{code}
checkRetryable(dbConn, e, "getOpenTxns");
throw new MetaException("Unable to select from transaction database, "
+ StringUtils.stringifyException(e));
{code}
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388770#comment-16388770
]
Eugene Koifman commented on HIVE-18886:
---
+1
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-18886:
--
Assignee: Gopal V
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18886:
---
Status: Patch Available (was: Open)
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18886:
---
Attachment: HIVE-18886.1.patch
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18886) ACID: NPE on unexplained mysql exceptions
[
https://issues.apache.org/jira/browse/HIVE-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18886:
---
Affects Version/s: 3.0.0
> ACID: NPE on unexplained mysql exceptions
> --
>
> Key: HIVE-18886
> URL: https://issues.apache.org/jira/browse/HIVE-18886
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18886.1.patch
>
>
> At 200+ sessions on a single HS2, the DbLock impl fails to propagate mysql
> exceptions
> {code}
> 2018-03-06T22:55:16,197 ERROR [HiveServer2-Background-Pool: Thread-12867]:
> ql.Driver (:()) - FAILED: Error in acquiring locks: null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.DatabaseProduct.isDeadlock(DatabaseProduct.java:56)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:2459)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:499)
> {code}
> {code}
> return e instanceof SQLTransactionRollbackException
> || ((dbProduct == MYSQL || dbProduct == POSTGRES || dbProduct ==
> SQLSERVER)
> && e.getSQLState().equals("40001"))
> || (dbProduct == POSTGRES && e.getSQLState().equals("40P01"))
> || (dbProduct == ORACLE && (e.getMessage().contains("deadlock
> detected")
> || e.getMessage().contains("can't serialize access for this
> transaction")));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388759#comment-16388759
]
Hive QA commented on HIVE-18864:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 432 unchanged - 0
fixed = 437 total (was 432) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9518/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9518/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9518/yetus/patch-asflicense-problems.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9518/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still
[jira] [Commented] (HIVE-18835) JDBC standalone jar download link in ambari
[ https://issues.apache.org/jira/browse/HIVE-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388749#comment-16388749 ] Thejas M Nair commented on HIVE-18835: -- [~mgergely] * Can you add log messages to list the path it tries to file from, in case of an error. If there is more than one, print the list of matches * Can you also add a UT. I am not very sure if this path would contain the jar in tests. In that case, maybe a simple UT to just verify the content type returned by this servlet might be useful. cc [~jxiang] in case you have any comments > JDBC standalone jar download link in ambari > --- > > Key: HIVE-18835 > URL: https://issues.apache.org/jira/browse/HIVE-18835 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-18835.patch > > > Let HS2 offer the file for download, so that Ambari can create link on it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Status: Patch Available (was: Open) > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[ https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18877: -- Attachment: HIVE-18877.1.patch > HiveSchemaTool.validateSchemaTables() should wrap a SQLException when > rethrowing > > > Key: HIVE-18877 > URL: https://issues.apache.org/jira/browse/HIVE-18877 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18877.1.patch > > > If schematool is run with the -verbose flag then it will print a stack trace > for an exception that occurs. If a SQLException is caught during > HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown > containing the text of the SQLException. If we instead throw a > HiveMetaException that wraps the SQLException then the stacktrace will help > with diagnosis of issues where the SQLException contains a generic error > text. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[ https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18034: Attachment: (was: HIVE-18034.5.patch) > Improving logging with HoS executors spend lots of time in GC > - > > Key: HIVE-18034 > URL: https://issues.apache.org/jira/browse/HIVE-18034 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18034.1.patch, HIVE-18034.2.patch, > HIVE-18034.3.patch, HIVE-18034.4.patch > > > There are times when Spark will spend lots of time doing GC. The Spark > History UI shows a bunch of red flags when too much time is spent in GC. It > would be nice if those warnings are propagated to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[ https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18034: Attachment: HIVE-18034.5.patch > Improving logging with HoS executors spend lots of time in GC > - > > Key: HIVE-18034 > URL: https://issues.apache.org/jira/browse/HIVE-18034 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18034.1.patch, HIVE-18034.2.patch, > HIVE-18034.3.patch, HIVE-18034.4.patch, HIVE-18034.5.patch > > > There are times when Spark will spend lots of time doing GC. The Spark > History UI shows a bunch of red flags when too much time is spent in GC. It > would be nice if those warnings are propagated to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388738#comment-16388738
]
Hive QA commented on HIVE-18693:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913230/HIVE-18693.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 97 failed/errored test(s), 13882 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_collect_set_unsupported]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_array_contains_wrong2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_coalesce]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_concat_ws_wrong1]
(batchId=95)
[jira] [Updated] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-17495:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-16520
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17495.1.patch, HIVE-17495.10.patch,
> HIVE-17495.10.patch, HIVE-17495.2.patch, HIVE-17495.3.patch,
> HIVE-17495.4.patch, HIVE-17495.5.patch, HIVE-17495.6.patch,
> HIVE-17495.7.patch, HIVE-17495.8.patch, HIVE-17495.9.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17629) CachedStore - wait for prewarm at use time, not init time
[ https://issues.apache.org/jira/browse/HIVE-17629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-17629: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore - wait for prewarm at use time, not init time > - > > Key: HIVE-17629 > URL: https://issues.apache.org/jira/browse/HIVE-17629 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17629.patch > > > Most of the changes are trivial, due to static changing to non-static. The > patch basically adds lifecycle management for shared cache, and uses it to > move prewarm to background thread, waiting for prewarm to happen at use time > not setConf, and to make init optional (if not called, CachedStore would > proxy the methods to rawStore, like it currently does in the ones not > implemented). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18056) CachedStore: Have a whitelist/blacklist config to allow selective caching of tables/partitions and allow read while prewarming
[ https://issues.apache.org/jira/browse/HIVE-18056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18056: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Have a whitelist/blacklist config to allow selective caching of > tables/partitions and allow read while prewarming > -- > > Key: HIVE-18056 > URL: https://issues.apache.org/jira/browse/HIVE-18056 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Daniel Dai >Priority: Major > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-18056.1.patch, HIVE-18056.2.patch, > HIVE-18056.3.patch, HIVE-18056.4.patch, HIVE-18056.5.patch, > HIVE-18056.6.patch, HIVE-18056.7.patch, HIVE-18056.8.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16848) NPE during CachedStore refresh
[
https://issues.apache.org/jira/browse/HIVE-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-16848:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-16520
> NPE during CachedStore refresh
> --
>
> Key: HIVE-16848
> URL: https://issues.apache.org/jira/browse/HIVE-16848
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-16848.1.patch, HIVE-16848.2.patch
>
>
> CachedStore refresh only happen once due to NPE during refresh of column
> statistics. ScheduledExecutorService canceled subsequent refreshes:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:458)
> at
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:348)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16779) CachedStore leak PersistenceManager resources
[ https://issues.apache.org/jira/browse/HIVE-16779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-16779: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore leak PersistenceManager resources > - > > Key: HIVE-16779 > URL: https://issues.apache.org/jira/browse/HIVE-16779 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-16779.1.patch, HIVE-16779.2.patch, > HIVE-16779.3.patch > > > See OOM when running CachedStore. We didn't shutdown rawstore in refresh > thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18847) CachedStore: Investigate TestCachedStore#testTableColStatsOps
[ https://issues.apache.org/jira/browse/HIVE-18847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-18847: --- Assignee: Vaibhav Gumashta > CachedStore: Investigate TestCachedStore#testTableColStatsOps > -- > > Key: HIVE-18847 > URL: https://issues.apache.org/jira/browse/HIVE-18847 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Currently commented out due to ObjectStore.updateTableColumnStatistics call > unable to persist stats to derby. Needs investigation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18847) CachedStore: Investigate TestCachedStore#testTableColStatsOps
[ https://issues.apache.org/jira/browse/HIVE-18847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18847: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Investigate TestCachedStore#testTableColStatsOps > -- > > Key: HIVE-18847 > URL: https://issues.apache.org/jira/browse/HIVE-18847 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Priority: Major > > Currently commented out due to ObjectStore.updateTableColumnStatistics call > unable to persist stats to derby. Needs investigation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18840) CachedStore: Prioritize loading of recently accessed tables during prewarm
[ https://issues.apache.org/jira/browse/HIVE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18840: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Prioritize loading of recently accessed tables during prewarm > -- > > Key: HIVE-18840 > URL: https://issues.apache.org/jira/browse/HIVE-18840 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > On clusters with large metadata, prewarming the cache can take several hours. > Now that CachedStore does not block on prewarm anymore (after HIVE-18264), we > should prioritize loading of recently accessed tables during prewarm. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18661: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17163) Microbenchmark for vector op processing
[ https://issues.apache.org/jira/browse/HIVE-17163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17163: Status: Patch Available (was: Open) > Microbenchmark for vector op processing > --- > > Key: HIVE-17163 > URL: https://issues.apache.org/jira/browse/HIVE-17163 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-17163.1.patch, HIVE-17163.2.patch > > > Add JMH based operator level benchmarks for vector operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388660#comment-16388660
]
Hive QA commented on HIVE-18693:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
46s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 283 unchanged - 3
fixed = 283 total (was 286) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} standalone-metastore: The patch generated 0 new +
539 unchanged - 1 fixed = 539 total (was 540) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9517/dev-support/hive-personality.sh
|
| git revision | master / abede8e |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9517/yetus/patch-mvninstall-hcatalog_streaming.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9517/yetus/patch-mvninstall-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9517/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9517/yetus/patch-asflicense-problems.txt
|
| modules | C: hcatalog/streaming ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9517/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Snapshot Isolation does not work for Micromanaged table when a insert
> transaction is aborted
>
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
[jira] [Assigned] (HIVE-17163) Microbenchmark for vector op processing
[ https://issues.apache.org/jira/browse/HIVE-17163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-17163: --- Assignee: Matt McCline (was: Prasanth Jayachandran) > Microbenchmark for vector op processing > --- > > Key: HIVE-17163 > URL: https://issues.apache.org/jira/browse/HIVE-17163 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-17163.1.patch, HIVE-17163.2.patch > > > Add JMH based operator level benchmarks for vector operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17163) Microbenchmark for vector op processing
[ https://issues.apache.org/jira/browse/HIVE-17163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17163: Attachment: HIVE-17163.2.patch > Microbenchmark for vector op processing > --- > > Key: HIVE-17163 > URL: https://issues.apache.org/jira/browse/HIVE-17163 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-17163.1.patch, HIVE-17163.2.patch > > > Add JMH based operator level benchmarks for vector operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388658#comment-16388658
]
Owen O'Malley commented on HIVE-17580:
--
{quote}I don't see a particular logically grouping of classes in storage-api
(for example, HiveDecimal is in storage-api, but the other types are in serde).
I think in the longer run we would need to reorganizing this in more consistent
modules anyways.
{quote}
I've tried, but not always succeeded. HiveDecimal is there, because it is used
by DecimalColumnVector. The other types aren't used by the other ColumnVectors.
It would be better if DecimalColumnVector did not use HiveDecimal, but that
would take a lot of work to fix.
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch,
> HIVE-17580.05-standalone-metastore.patch,
> HIVE-17580.06-standalone-metastore.patch,
> HIVE-17580.07-standalone-metastore.patch,
> HIVE-17580.08-standalone-metastore.patch,
> HIVE-17580.09-standalone-metastore.patch,
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388657#comment-16388657 ] Sahil Takiar commented on HIVE-18883: - [~szita], [~pvary] any tips on how to do this? > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-18883: --- > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388655#comment-16388655
]
Owen O'Malley commented on HIVE-17580:
--
{quote}The other option suggested of moving ObjectInspector to
standalone-metastore sounds weird to me since it has got nothing to do with
metastore. **
{quote}
It has even less to do with storage-api, which is precisely why I don't want it
there. Even worse, it moves really high up the release tree:
# storage-api
# ORC
# metastore
# hive
Moving things into storage-api that don't need to be there is a big cost.
Moving it to metastore that actually does need the enum makes at least some
sense. Nothing in storage-api or ORC need or want that enum.
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch,
> HIVE-17580.05-standalone-metastore.patch,
> HIVE-17580.06-standalone-metastore.patch,
> HIVE-17580.07-standalone-metastore.patch,
> HIVE-17580.08-standalone-metastore.patch,
> HIVE-17580.09-standalone-metastore.patch,
> HIVE-17580.092-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18835) JDBC standalone jar download link in ambari
[
https://issues.apache.org/jira/browse/HIVE-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388635#comment-16388635
]
Hive QA commented on HIVE-18835:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913226/HIVE-18835.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 13483 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18881:
---
Labels: noob (was: )
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Labels: noob
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18881:
---
Description:
[https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
{code:java}
LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
{code}
# All the other logging in this class is _debug_ or _error_ level logging.
This should be _debug_ as well
# Remove tab characters to allow splitting on tabs in any kind of
tab-separated file of log lines
# Use SLF4J parameterized logging
was:
https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101
{code}
LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
{code}
# All the other logging in this class is _debug_ or _error_ level logging.
This should be debug
# Remove tab characters to allow splitting on tabs in any kind of tab-separated
file of log lines
# Use SLF4J parameterized logging
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388613#comment-16388613
]
Daniel Dai commented on HIVE-18778:
---
[~ngangam], are you able working on this?
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778.1.patch, HIVE-18778.2.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18879) Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath
[ https://issues.apache.org/jira/browse/HIVE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18879: -- Target Version/s: 3.0.0 > Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in > classpath > -- > > Key: HIVE-18879 > URL: https://issues.apache.org/jira/browse/HIVE-18879 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18879.1.patch > > > This is a follow up of HIVE-18789. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18879) Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath
[ https://issues.apache.org/jira/browse/HIVE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18879: -- Attachment: HIVE-18879.1.patch > Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in > classpath > -- > > Key: HIVE-18879 > URL: https://issues.apache.org/jira/browse/HIVE-18879 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18879.1.patch > > > This is a follow up of HIVE-18789. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18879) Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath
[ https://issues.apache.org/jira/browse/HIVE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18879: -- Status: Patch Available (was: Open) > Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in > classpath > -- > > Key: HIVE-18879 > URL: https://issues.apache.org/jira/browse/HIVE-18879 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18879.1.patch > > > This is a follow up of HIVE-18789. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18879) Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath
[ https://issues.apache.org/jira/browse/HIVE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-18879: - Assignee: Daniel Dai > Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in > classpath > -- > > Key: HIVE-18879 > URL: https://issues.apache.org/jira/browse/HIVE-18879 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > This is a follow up of HIVE-18789. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
