[jira] [Commented] (HIVE-22534) ACID: Improve Compactor thread logging
[
https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988557#comment-16988557
]
Hive QA commented on HIVE-22534:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 48 unchanged - 4
fixed = 48 total (was 52) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19750/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19750/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Improve Compactor thread logging
> --
>
> Key: HIVE-22534
> URL: https://issues.apache.org/jira/browse/HIVE-22534
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch,
> HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch,
> HIVE-22534.06.patch, HIVE-22534.07.patch
>
>
> Make sure that it is easy to find issues when one of the compactor thread
> fails.
> Maybe:
> * MDC - with iteration / threadname - so we can easily grep the logs for a
> given run
> * MDC with table/partition data on which the worker is working
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988539#comment-16988539
]
Hive QA commented on HIVE-22580:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987469/HIVE-22580.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17757 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19749/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19749/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19749/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987469 - PreCommit-HIVE-Build
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-22580.patch
>
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:604)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:560)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8095)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy38.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3139)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3103)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy39.openTxn(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:250)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:234)
> at org.apache.hadoop.hive.ql.Driver.openTransaction(Driver.java:915)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:640)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1387)
> at
[jira] [Commented] (HIVE-22473) Avoid reading table as ACID when table name is starting with "delta", but table is not transactional
[
https://issues.apache.org/jira/browse/HIVE-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988522#comment-16988522
]
Aditya Shah commented on HIVE-22473:
[~lpinter], I faced the same issue after the fix. I think, we should add a
similar check for table to be acid in BISplitStategy's getSplit function?
> Avoid reading table as ACID when table name is starting with "delta", but
> table is not transactional
>
>
> Key: HIVE-22473
> URL: https://issues.apache.org/jira/browse/HIVE-22473
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22473.01.patch, HIVE-22473.02.patch,
> HIVE-22473.03.patch, HIVE-22473.04.patch
>
>
> {code:sql}
> create table delta_result (a int) stored as orc
> tblproperties('transactional'='false');
> insert into delta_result select 1;
> select * from delta_result;
> {code}
> The above query will result in the following exception:
> 2019-11-08T13:49:05,780 WARN [HiveServer2-Handler-Pool: Thread-7906]
> thrift.ThriftCLIService: Error fetching results:2019-11-08T13:49:05,780 WARN
> [HiveServer2-Handler-Pool: Thread-7906] thrift.ThriftCLIService: Error
> fetching results:org.apache.hive.service.cli.HiveSQLException:
> java.io.IOException: java.lang.RuntimeException: ORC split generation failed
> with exception: java.lang.StringIndexOutOfBoundsException: String index out
> of range: -1 at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:481)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:331)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:801)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_211] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]Caused
> by: java.io.IOException: java.lang.RuntimeException: ORC split generation
> failed with exception: java.lang.StringIndexOutOfBoundsException: String
> index out of range: -1 at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2142)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] ... 13 moreCaused by:
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016)
>
[jira] [Commented] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988497#comment-16988497
]
Hive QA commented on HIVE-22580:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 18 unchanged - 1 fixed = 19 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19749/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19749/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19749/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-22580.patch
>
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at
[jira] [Updated] (HIVE-22508) KeyWrapperComparator throws exception
[ https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22508: -- Status: Patch Available (was: Open) > KeyWrapperComparator throws exception > - > > Key: HIVE-22508 > URL: https://issues.apache.org/jira/browse/HIVE-22508 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch, > HIVE-22508.1.patch > > > TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a > copied key should be compared. > The current implementation uses the standard object inspectors for all > KeyWrapper instances. However when comparing untouched KeyWrappers the key > object inspector should be used which can be extracted form > Operator.inputObjectInspectors during initialization of the key's > ExprNodeEvaluator. > This can cause a ClassCastException when the comparator is used collections > like TreeSet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22508) KeyWrapperComparator throws exception
[ https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22508: -- Status: Open (was: Patch Available) > KeyWrapperComparator throws exception > - > > Key: HIVE-22508 > URL: https://issues.apache.org/jira/browse/HIVE-22508 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch, > HIVE-22508.1.patch > > > TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a > copied key should be compared. > The current implementation uses the standard object inspectors for all > KeyWrapper instances. However when comparing untouched KeyWrappers the key > object inspector should be used which can be extracted form > Operator.inputObjectInspectors during initialization of the key's > ExprNodeEvaluator. > This can cause a ClassCastException when the comparator is used collections > like TreeSet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22508) KeyWrapperComparator throws exception
[ https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22508: -- Attachment: HIVE-22508.1.patch > KeyWrapperComparator throws exception > - > > Key: HIVE-22508 > URL: https://issues.apache.org/jira/browse/HIVE-22508 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch, > HIVE-22508.1.patch > > > TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a > copied key should be compared. > The current implementation uses the standard object inspectors for all > KeyWrapper instances. However when comparing untouched KeyWrappers the key > object inspector should be used which can be extracted form > Operator.inputObjectInspectors during initialization of the key's > ExprNodeEvaluator. > This can cause a ClassCastException when the comparator is used collections > like TreeSet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?focusedWorklogId=354052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354052 ] ASF GitHub Bot logged work on HIVE-22408: - Author: ASF GitHub Bot Created on: 05/Dec/19 05:42 Start Date: 05/Dec/19 05:42 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #860: HIVE-22408 The fix for CALCITE-2991 creates wrong results on edge case URL: https://github.com/apache/hive/pull/860 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354052) Remaining Estimate: 0h Time Spent: 10m > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22408: -- Labels: pull-request-available (was: ) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988486#comment-16988486 ] Vineet Garg commented on HIVE-22408: [~jcamachorodriguez] can you take a look? https://github.com/apache/hive/pull/860 > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22517) Sysdb related qtests also output the sysdb sql commands to q.out
[
https://issues.apache.org/jira/browse/HIVE-22517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988483#comment-16988483
]
Hive QA commented on HIVE-22517:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987456/HIVE-22517.02.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17757 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19748/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19748/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19748/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987456 - PreCommit-HIVE-Build
> Sysdb related qtests also output the sysdb sql commands to q.out
>
>
> Key: HIVE-22517
> URL: https://issues.apache.org/jira/browse/HIVE-22517
> Project: Hive
> Issue Type: Improvement
> Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22517.01.patch, HIVE-22517.02.patch,
> HIVE-22517.02.patch, HIVE-22517.02.patch, HIVE-22517.02.patch,
> HIVE-22517.02.patch, HIVE-22517.02.patch, HIVE-22517.02.patch
>
>
> it would be better to not have it on the outputs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988449#comment-16988449
]
Hive QA commented on HIVE-22408:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987451/HIVE-22408.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17757 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19747/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19747/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19747/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987451 - PreCommit-HIVE-Build
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch
>
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988414#comment-16988414
]
Hive QA commented on HIVE-22408:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 2 unchanged - 22
fixed = 2 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19747/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19747/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch
>
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[
https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988399#comment-16988399
]
Hive QA commented on HIVE-22579:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987446/HIVE-22579.01.branch-2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19746/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19746/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19746/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-12-05 02:34:47.581
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19746/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-2 ]]
+ [[ -d apache-github-branch-2-source ]]
+ [[ ! -d apache-github-branch-2-source/.git ]]
+ [[ ! -d apache-github-branch-2-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-12-05 02:34:47.679
+ cd apache-github-branch-2-source
+ git fetch origin
>From https://github.com/apache/hive
7534f82..de0a7ec branch-1 -> origin/branch-1
9bcdb54..6002c51 branch-1.2 -> origin/branch-1.2
292a98f..0359921 branch-2.1 -> origin/branch-2.1
b148507..67f9139 branch-2.2 -> origin/branch-2.2
f90975a..f55ee60 branch-3 -> origin/branch-3
a354bed..0ecbd12 branch-3.0 -> origin/branch-3.0
909c1dc..eb4d7c3 branch-3.1 -> origin/branch-3.1
305e710..1ef05ef master -> origin/master
e59fdf9..3638231 storage-branch-2.7 -> origin/storage-branch-2.7
* [new tag] rel/storage-release-2.7.1 -> rel/storage-release-2.7.1
+ git reset --hard HEAD
HEAD is now at a4a6101 HIVE-22249: Support Parquet through HCatalog (Jay
Green-Stevens via Peter Vary)
+ git clean -f -d
+ git checkout branch-2
Already on 'branch-2'
Your branch is up-to-date with 'origin/branch-2'.
+ git reset --hard origin/branch-2
HEAD is now at a4a6101 HIVE-22249: Support Parquet through HCatalog (Jay
Green-Stevens via Peter Vary)
+ git merge --ff-only origin/branch-2
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-12-05 02:35:11.141
+ rm -rf ../yetus_PreCommit-HIVE-Build-19746
+ mkdir ../yetus_PreCommit-HIVE-Build-19746
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19746
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19746/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-branch-2-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-branch-2-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer : Classpath
>> /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) :
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) :
[jira] [Commented] (HIVE-22489) Reduce Sink operator should order nulls by parameter
[
https://issues.apache.org/jira/browse/HIVE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988396#comment-16988396
]
Hive QA commented on HIVE-22489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987445/HIVE-22489.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1320 failed/errored test(s), 17758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=293)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=301)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=301)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=301)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=304)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguous_col]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby2]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join_pkfk]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join27]
(batchId=100)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binarysortable_1]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_SortUnionTransposeRule]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_annotate_stats_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_gby2_map_multi_distinct]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_groupby3_noskew_multi_distinct]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join0]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_pruner_multiple_children]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine2] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[complex_alias]
(batchId=19)
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-22573: --- Attachment: HIVE-22573.3.patch > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.2.patch, HIVE-22573.3.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22489) Reduce Sink operator should order nulls by parameter
[
https://issues.apache.org/jira/browse/HIVE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988378#comment-16988378
]
Hive QA commented on HIVE-22489:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} serde: The patch generated 3 new + 87 unchanged - 0
fixed = 90 total (was 87) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 2 new + 579 unchanged - 0
fixed = 581 total (was 579) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19745/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19745/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19745/yetus/diff-checkstyle-ql.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19745/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce Sink operator should order nulls by parameter
> -
>
> Key: HIVE-22489
> URL: https://issues.apache.org/jira/browse/HIVE-22489
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22489.1.patch, HIVE-22489.2.patch,
> HIVE-22489.3.patch, HIVE-22489.3.patch
>
>
> When the property hive.default.nulls.last is set to true and no null order is
> explicitly specified in the ORDER BY clause of the query null ordering should
> be NULLS LAST.
> But some of the Reduce Sink operators still orders null first.
> {code}
> SET
[jira] [Commented] (HIVE-22574) Hive on Spark Jobs Failed With Error: Client closed before SASL negotiation finished
[
https://issues.apache.org/jira/browse/HIVE-22574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988365#comment-16988365
]
lilizhi commented on HIVE-22574:
The error is in org.apache.hive.spark.client.rpc.SaslHandler：
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
dispose(); // Do retry here ?
super.channelInactive(ctx);
}
> Hive on Spark Jobs Failed With Error: Client closed before SASL negotiation
> finished
>
>
> Key: HIVE-22574
> URL: https://issues.apache.org/jira/browse/HIVE-22574
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Spark
>Affects Versions: 1.2.2
> Environment: spark : spark-1.6.0-cdh5.16.1
> hive : hive-1.1.0-cdh5.16.1
> hadoop : hadoop-2.6.0-cdh5.16.2
>Reporter: lilizhi
>Priority: Minor
>
> Hive on Spark job failed with below error message, this happens only when
> cluster is busy:
> Excerpt from exception stack from one of the YARN applications.
> 19/12/02 16:19:00 ERROR yarn.ApplicationMaster: Uncaught exception:
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException:
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL
> negotiation finished.
> at
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:455)
> at
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java
> 90)
> Is it helpful with retry?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[
https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988359#comment-16988359
]
Hive QA commented on HIVE-22502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987144/HIVE-22502.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17757 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook]
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19744/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19744/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987144 - PreCommit-HIVE-Build
> ConcurrentModificationException in TriggerValidatorRunnable stops trigger
> processing
>
>
> Key: HIVE-22502
> URL: https://issues.apache.org/jira/browse/HIVE-22502
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22502.1.patch
>
>
> An other thread is modifying the list the contains the sessions while
> TriggerValidatorRunnable is traversing it. This causes the
> TriggerValidatorRunnable thread to die and triggers are no longer properly
> monitored.
>
> {code:java}
> <12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught exception.<12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught
> exception.java.util.ConcurrentModificationException at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at
> java.util.LinkedList$ListItr.next(LinkedList.java:888) at
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) at
> org.apache.hadoop.hive.ql.exec.tez.TriggerValidatorRunnable.run(TriggerValidatorRunnable.java:49)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-22573: --- Attachment: HIVE-22573.2.patch > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.2.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-22573: --- Attachment: (was: HIVE-22573.01.patch) > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.2.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Mani updated HIVE-22573: --- Attachment: HIVE-22573.01.patch > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.01.patch, HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22502) ConcurrentModificationException in TriggerValidatorRunnable stops trigger processing
[
https://issues.apache.org/jira/browse/HIVE-22502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988315#comment-16988315
]
Hive QA commented on HIVE-22502:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19744/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19744/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ConcurrentModificationException in TriggerValidatorRunnable stops trigger
> processing
>
>
> Key: HIVE-22502
> URL: https://issues.apache.org/jira/browse/HIVE-22502
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Attachments: HIVE-22502.1.patch
>
>
> An other thread is modifying the list the contains the sessions while
> TriggerValidatorRunnable is traversing it. This causes the
> TriggerValidatorRunnable thread to die and triggers are no longer properly
> monitored.
>
> {code:java}
> <12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught exception.<12>1 2019-11-14T00:31:12.187Z
> hiveserver2-0.hiveserver2-service.compute-1572769905-6965.svc.cluster.local
> hiveserver2 1 fa2f30b6-ffb3-11e9-93ba-0a257c2413a2 [mdc@18060
> class="tez.TriggerValidatorRunnable" level="WARN" thread="TriggerValidator"]
> TriggerValidatorRunnable caught
> exception.java.util.ConcurrentModificationException at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at
> java.util.LinkedList$ListItr.next(LinkedList.java:888) at
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) at
>
[jira] [Commented] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988296#comment-16988296
]
Hive QA commented on HIVE-21266:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987478/HIVE-21266.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17759 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19743/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19743/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19743/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987478 - PreCommit-HIVE-Build
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch, HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22355) Beeline should not prompt for hive user and password when authentication is NONE
[ https://issues.apache.org/jira/browse/HIVE-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988294#comment-16988294 ] Thejas Nair commented on HIVE-22355: Why is username defaulting to "hive" ? IMO, it should behave like "hive" cli used to, which is to pick current unix username. HIVE-20734 made the hive redirect to beeline use that. cc [~maheshk114] [~ngangam] > Beeline should not prompt for hive user and password when authentication is > NONE > > > Key: HIVE-22355 > URL: https://issues.apache.org/jira/browse/HIVE-22355 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Major > Attachments: HIVE-22355.1.patch, HIVE-22355.2.patch, > HIVE-22355.3.patch > > > Beeline - without adding the jdbc url - prompts for username and password in > case hive.server2.authentication=NONE, which is possibly pointless and can be > misleading for users as any input is accepted. > In addition, Sqoop has dropped hive cli recently in favor of beeline and if > there is no authentication set in Hive, Sqoop fails to connect as the process > stops waiting for the user/password input. > I think it would be nice to check the auth type "NONE" before reading unused > inputs from the console before this point: > https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/Commands.java#L1641 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22573) HMS filter metadata using Ranger plugin
[ https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-22573: - Assignee: Ramesh Mani (was: Sam An) > HMS filter metadata using Ranger plugin > --- > > Key: HIVE-22573 > URL: https://issues.apache.org/jira/browse/HIVE-22573 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Ramesh Mani >Priority: Major > Attachments: HIVE-22573.patch > > > HMS needs to filter out rows based on Ranger plugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988275#comment-16988275
]
Hive QA commented on HIVE-21266:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 1 new + 758 unchanged - 2
fixed = 759 total (was 760) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19743/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19743/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19743/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch, HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988257#comment-16988257
]
Hive QA commented on HIVE-16690:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987438/HIVE-16690.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17757 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19742/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19742/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19742/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987438 - PreCommit-HIVE-Build
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[
https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988221#comment-16988221
]
Hive QA commented on HIVE-16690:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
47s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 3 new + 37 unchanged - 0 fixed
= 40 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19742/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19742/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19742/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configure Tez cartesian product edge based on LLAP cluster size
> ---
>
> Key: HIVE-16690
> URL: https://issues.apache.org/jira/browse/HIVE-16690
> Project: Hive
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch,
> HIVE-16690.addendum.patch
>
>
> In HIVE-14731 we are using default value for target parallelism of fair
> cartesian product edge. Ideally this should be set according to cluster size.
> In case of LLAP it's pretty easy to get cluster size, i.e., number of
> executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988191#comment-16988191 ] Oliver Draese commented on HIVE-22572: -- UnitTests to be added via https://issues.apache.org/jira/browse/HIVE-22581 > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22581) Add tests for dynamic semijoins reduction
[ https://issues.apache.org/jira/browse/HIVE-22581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22581: > Add tests for dynamic semijoins reduction > - > > Key: HIVE-22581 > URL: https://issues.apache.org/jira/browse/HIVE-22581 > Project: Hive > Issue Type: Test > Components: Query Planning >Reporter: Oliver Draese >Assignee: Jesus Camacho Rodriguez >Priority: Major > > There don't seem to be tests for the TezCompiler. A test suite should be > added and a test for the error scenario of HIVE-22572 should be implemented > there. > > This is a follow-up action for: > https://issues.apache.org/jira/browse/HIVE-22572 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location
[
https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988187#comment-16988187
]
Hive QA commented on HIVE-22548:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987425/HIVE-22548.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17758 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=277)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=277)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentLineage (batchId=289)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=289)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=289)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=289)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=289)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19741/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19741/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19741/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987425 - PreCommit-HIVE-Build
> Optimise Utilities.removeTempOrDuplicateFiles when moving files to final
> location
> -
>
> Key: HIVE-22548
> URL: https://issues.apache.org/jira/browse/HIVE-22548
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22548.01.patch
>
>
> {{Utilities.removeTempOrDuplicateFiles}}
> is very slow with cloud storage, as it executes {{listStatus}} twice and also
> runs in single threaded mode.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1629
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988183#comment-16988183 ] Jesus Camacho Rodriguez commented on HIVE-22572: +1 (pending tests) > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location
[
https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988128#comment-16988128
]
Hive QA commented on HIVE-22548:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
37s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 1 new + 105 unchanged - 0
fixed = 106 total (was 105) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19741/dev-support/hive-personality.sh
|
| git revision | master / 1ef05ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19741/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19741/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise Utilities.removeTempOrDuplicateFiles when moving files to final
> location
> -
>
> Key: HIVE-22548
> URL: https://issues.apache.org/jira/browse/HIVE-22548
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22548.01.patch
>
>
> {{Utilities.removeTempOrDuplicateFiles}}
> is very slow with cloud storage, as it executes {{listStatus}} twice and also
> runs in single threaded mode.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1629
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988095#comment-16988095
]
Hive QA commented on HIVE-22573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987414/HIVE-22573.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_acid_no_masking]
(batchId=26)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19740/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19740/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19740/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987414 - PreCommit-HIVE-Build
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988093#comment-16988093 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.1.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22499) LLAP: Add an EncodedReaderOptions to extend ORC impl for options
[ https://issues.apache.org/jira/browse/HIVE-22499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22499: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Mustafa! > LLAP: Add an EncodedReaderOptions to extend ORC impl for options > > > Key: HIVE-22499 > URL: https://issues.apache.org/jira/browse/HIVE-22499 > Project: Hive > Issue Type: Bug > Components: llap, ORC >Reporter: Gopal Vijayaraghavan >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22499.2.patch, HIVE-22499.2.patch, > HIVE-22499.3.patch, HIVE-22499.4.patch, HIVE-22499.5.patch, > HIVE-22499.6.patch, HIVE-22499.7.patch, HIVE-22499.WIP.patch, HIVE-22499.patch > > Time Spent: 10m > Remaining Estimate: 0h > > ORC-570 is an ABI change to the way getFileSystem() by adding an another > exception to the implementation. > To accept and use that change requires waiting for an ORC release, while this > patch serves the same purpose though falls back for a retry for > FileSystem.get() in case the supplier fails at runtime. > Also as a side-note, the FS.get() call is always used in the cases where the > file is not being read from a cache such as EncodedOrcFile (so the upstream > API change might be overkill). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22573) HMS filter metadata using Ranger plugin
[
https://issues.apache.org/jira/browse/HIVE-22573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988057#comment-16988057
]
Hive QA commented on HIVE-22573:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
39s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 218 new + 145 unchanged - 3
fixed = 363 total (was 148) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
47s{color} | {color:red} ql generated 1 new + 1532 unchanged - 0 fixed = 1533
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Write to static field
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.hiveAuthorizer
from instance method
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(PreEventContext)
At HiveMetaStoreAuthorizer.java:from instance method
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(PreEventContext)
At HiveMetaStoreAuthorizer.java:[line 102] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19740/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19740/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19740/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19740/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HMS filter metadata using Ranger plugin
> ---
>
> Key: HIVE-22573
> URL: https://issues.apache.org/jira/browse/HIVE-22573
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22573.patch
>
>
> HMS needs to filter out rows based on Ranger plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988020#comment-16988020
]
Hive QA commented on HIVE-22572:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987416/HIVE-22572.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook]
(batchId=296)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity
(batchId=366)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19739/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987416 - PreCommit-HIVE-Build
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22395) Add ability to read Druid metastore password from jceks
[ https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987998#comment-16987998 ] Ashutosh Chauhan commented on HIVE-22395: - +1 > Add ability to read Druid metastore password from jceks > --- > > Key: HIVE-22395 > URL: https://issues.apache.org/jira/browse/HIVE-22395 > Project: Hive > Issue Type: Bug >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch, > HIVE-22395.2.patch, HIVE-22395.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22395) Add ability to read Druid metastore password from jceks
[ https://issues.apache.org/jira/browse/HIVE-22395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-22395: Attachment: HIVE-22395.2.patch > Add ability to read Druid metastore password from jceks > --- > > Key: HIVE-22395 > URL: https://issues.apache.org/jira/browse/HIVE-22395 > Project: Hive > Issue Type: Bug >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22395.1.patch, HIVE-22395.2.patch, > HIVE-22395.2.patch, HIVE-22395.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987984#comment-16987984
]
Hive QA commented on HIVE-22572:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
53s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19739/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19739/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
>
[jira] [Commented] (HIVE-22355) Beeline should not prompt for hive user and password when authentication is NONE
[ https://issues.apache.org/jira/browse/HIVE-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987964#comment-16987964 ] Mate Juhasz commented on HIVE-22355: Are we allowed to set the username by default to "hive" (when auth is "NONE") if its not specified among the connection params or in the url? Otherwise the user would connect as "anonymous", without any read/write privileges > Beeline should not prompt for hive user and password when authentication is > NONE > > > Key: HIVE-22355 > URL: https://issues.apache.org/jira/browse/HIVE-22355 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Mate Juhasz >Assignee: Mate Juhasz >Priority: Major > Attachments: HIVE-22355.1.patch, HIVE-22355.2.patch, > HIVE-22355.3.patch > > > Beeline - without adding the jdbc url - prompts for username and password in > case hive.server2.authentication=NONE, which is possibly pointless and can be > misleading for users as any input is accepted. > In addition, Sqoop has dropped hive cli recently in favor of beeline and if > there is no authentication set in Hive, Sqoop fails to connect as the process > stops waiting for the user/password input. > I think it would be nice to check the auth type "NONE" before reading unused > inputs from the console before this point: > https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/Commands.java#L1641 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[
https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987961#comment-16987961
]
Hive QA commented on HIVE-22539:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987396/HIVE-22539.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query5]
(batchId=28)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19738/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19738/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987396 - PreCommit-HIVE-Build
> HiveServer2 SPNEGO authentication should skip if authorization header is empty
> --
>
> Key: HIVE-22539
> URL: https://issues.apache.org/jira/browse/HIVE-22539
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch
>
>
> Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos
> before checking header. This can be checked up front to avoid doing any
> Kerberos related work if the header is empty. This is helpful in a lot of
> cases since typically the first request is empty with the client waiting for
> a 401 before returning the Authorization header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22570) Review of ExprNodeDesc.java
[ https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22570: -- Status: Patch Available (was: Open) > Review of ExprNodeDesc.java > --- > > Key: HIVE-22570 > URL: https://issues.apache.org/jira/browse/HIVE-22570 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22570) Review of ExprNodeDesc.java
[ https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22570: -- Attachment: HIVE-22570.1.patch > Review of ExprNodeDesc.java > --- > > Key: HIVE-22570 > URL: https://issues.apache.org/jira/browse/HIVE-22570 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22570) Review of ExprNodeDesc.java
[ https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22570: -- Status: Open (was: Patch Available) > Review of ExprNodeDesc.java > --- > > Key: HIVE-22570 > URL: https://issues.apache.org/jira/browse/HIVE-22570 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22570.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987929#comment-16987929
]
David Mollitor commented on HIVE-22569:
---
OK. Ya. This is worthwhile. One place in the code passes in a 'null' value
(and caused unit tests to fail) and another place passes in an empty list. I
have changed the patch so that all the callers now pass an empty list (and not
a null value). I also added checks in the constructor of
{{PrunedPartitionList}} to fail if anyone else tries to pass in a null value
again.
I also noticed, in the two constructors, that the {{ppListKey}} is an optional
field, it can be set to null. I changed the value of {{ppListKey}} to
therefore be an {{Optional}}. In doing this, I noticed that several of the
places that request the key do not ever consider that the field can be null and
are at risk for generating a NPE. I won't fix all of that now, but making the
data type {{Optional}} will at least force folks to think about it later.
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[
https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987930#comment-16987930
]
Kevin Risden commented on HIVE-22539:
-
Checkstyle is complaining about package-info.java. That file doesn't look to be
there for any other test packages?
{code:java}
./service/src/test/org/apache/hive/service/cli/thrift/ThriftHttpServletTest.java:0::
warning: Missing package-info.java file.
{code}
> HiveServer2 SPNEGO authentication should skip if authorization header is empty
> --
>
> Key: HIVE-22539
> URL: https://issues.apache.org/jira/browse/HIVE-22539
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch
>
>
> Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos
> before checking header. This can be checked up front to avoid doing any
> Kerberos related work if the header is empty. This is helpful in a lot of
> cases since typically the first request is empty with the client waiting for
> a 401 before returning the Authorization header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[ https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22578 started by Aron Hamvas. -- > CBO genOPTree is not failsafe for CTAS and VIEW statements > -- > > Key: HIVE-22578 > URL: https://issues.apache.org/jira/browse/HIVE-22578 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > > If CBO fails during OT generation, it is supposed to skip CBO and fall back > to SemanticAnalyzer generating the OT for the original AST. However, for CTAS > and VIEW statements, the original AST is discarded and replaced with the new > AST. > If OT generation for the new AST fails with a SemanticException, the > compilation is destined to fail completely since we cannot fall back to the > original and possibly valid AST. > Also, the SemanticException sometimes gets hidden and misjudged as a missing > column statistics issue due to bad error handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22569:
--
Attachment: HIVE-22569.2.patch
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22569.2.patch, HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[
https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987925#comment-16987925
]
Hive QA commented on HIVE-22539:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} service: The patch generated 1 new + 25 unchanged - 6
fixed = 26 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} service generated 0 new + 48 unchanged - 1 fixed =
48 total (was 49) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19738/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19738/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19738/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2 SPNEGO authentication should skip if authorization header is empty
> --
>
> Key: HIVE-22539
> URL: https://issues.apache.org/jira/browse/HIVE-22539
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch
>
>
> Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos
> before checking header. This can be checked up front to avoid doing any
> Kerberos related work if the header is empty. This is helpful in a lot of
> cases since typically the first request is empty with the client waiting for
> a 401 before returning the Authorization header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Status: Open (was: Patch Available) > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Status: Patch Available (was: Open) > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22571: -- Attachment: HIVE-22571.1.patch > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch, HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22570) Review of ExprNodeDesc.java
[
https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987908#comment-16987908
]
Hive QA commented on HIVE-22570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987394/HIVE-22570.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=112)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19737/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19737/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19737/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987394 - PreCommit-HIVE-Build
> Review of ExprNodeDesc.java
> ---
>
> Key: HIVE-22570
> URL: https://issues.apache.org/jira/browse/HIVE-22570
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22570.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21266:
-
Attachment: (was: HIVE-21234.4.patch)
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch, HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21266:
-
Status: Open (was: Patch Available)
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch, HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21266:
-
Attachment: HIVE-21266.04.patch
Status: Patch Available (was: Open)
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch, HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21266:
-
Attachment: HIVE-21234.4.patch
Status: Patch Available (was: Open)
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21266:
-
Status: Open (was: Patch Available)
> Don't run cleaner if compaction is skipped (issue with single delta file)
> -
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch,
> HIVE-21266.02.patch, HIVE-21266.03.patch, HIVE-21266.04.patch,
> HIVE-21266.04.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Status: Patch Available (was: Open)
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22097) Incompatible java.util.ArrayList for java 11
[
https://issues.apache.org/jira/browse/HIVE-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22097:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Incompatible java.util.ArrayList for java 11
>
>
> Key: HIVE-22097
> URL: https://issues.apache.org/jira/browse/HIVE-22097
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Yuming Wang
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22097.1.branch-3.1.patch,
> HIVE-22097.1.branch-3.patch, HIVE-22097.1.patch, JDK1.8.png, JDK11.png
>
>
> {noformat}
> export JAVA_HOME=/usr/lib/jdk-11.0.3
> export PATH=${JAVA_HOME}/bin:${PATH}
> hive> create table t(id int);
> Time taken: 0.035 seconds
> hive> insert into t values(1);
> Query ID = root_20190811155400_7c0e0494-eecb-4c54-a9fd-942ab52a0794
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:390)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:235)
> at
> org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:280)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:587)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:579)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:357)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2317)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1969)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1636)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1396)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1390)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:162)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:223)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:838)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:777)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:696)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.NoSuchFieldException: parentOffset
> at java.base/java.lang.Class.getDeclaredField(Class.java:2412)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.(SerializationUtilities.java:384)
> ... 29 more
> Job Submission failed with exception
> 'java.lang.RuntimeException(java.lang.NoSuchFieldException: parentOffset)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.lang.NoSuchFieldException:
> parentOffset
> {noformat}
> The reason is Java removed {{parentOffset}}:
> !JDK1.8.png!
> !JDK11.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20972) Enable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
[ https://issues.apache.org/jira/browse/HIVE-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20972: Attachment: HIVE-20972.02.patch > Enable TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit] > -- > > Key: HIVE-20972 > URL: https://issues.apache.org/jira/browse/HIVE-20972 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20972.01.patch, HIVE-20972.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.07.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch, HIVE-22534.07.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.09.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22570) Review of ExprNodeDesc.java
[
https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987864#comment-16987864
]
Hive QA commented on HIVE-22570:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19737/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19737/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of ExprNodeDesc.java
> ---
>
> Key: HIVE-22570
> URL: https://issues.apache.org/jira/browse/HIVE-22570
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22570.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22576) tez.queue.name setting is invalid when no default queue is set on yarn
[
https://issues.apache.org/jira/browse/HIVE-22576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hezhang updated HIVE-22576:
---
Priority: Critical (was: Major)
> tez.queue.name setting is invalid when no default queue is set on yarn
> ---
>
> Key: HIVE-22576
> URL: https://issues.apache.org/jira/browse/HIVE-22576
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.1, 2.3.2
>Reporter: hezhang
>Assignee: hezhang
>Priority: Critical
> Fix For: 2.3.2
>
> Attachments: HIVE-22576.1.patch, image-2019-12-04-15-49-03-475.png
>
>
> h2. hive cli started:
> hive> set tez.queue.name=prod;
> hive>
> > select count(*) from test;
> Query ID = hadoop_20191204152859_d9f7fff6-c0a1-4984-9f56-dd958419992b
> Total jobs = 1
> Launching Job 1 out of 1
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> err stack
>
> {code:java}
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:163)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:144)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> ~[hive-cli-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_191]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_191]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> ~[hadoop-common-2.7.3.jar:?]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> ~[hadoop-common-2.7.3.jar:?]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_191]
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> ~[?:1.8.0_191]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:158)
> ~[hive-exec-2.3.2.jar:2.3.2]
> ... 20 more
> Caused by: org.apache.tez.dag.api.TezException:
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
> application_1575443228569_0002 to YARN : Application
> application_1575443228569_0002 submitted by user hadoop to unknown queue:
> default
> at org.apache.tez.client.TezClient.start(TezClient.java:492)
> ~[tez-api-0.8.5.jar:0.8.5]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:376)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:87)
>
[jira] [Updated] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22580:
--
Status: Patch Available (was: Open)
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-22580.patch
>
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:604)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:560)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8095)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy38.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3139)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3103)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy39.openTxn(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:250)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:234)
> at org.apache.hadoop.hive.ql.Driver.openTransaction(Driver.java:915)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:640)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1387)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1376)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.dropTable(TestDbTxnManager2.java:974)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLockBlockedBy(TestDbTxnManager2.java:396)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>
[jira] [Updated] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22580:
--
Attachment: HIVE-22580.patch
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-22580.patch
>
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:604)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:560)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8095)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy38.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3139)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3103)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy39.openTxn(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:250)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:234)
> at org.apache.hadoop.hive.ql.Driver.openTransaction(Driver.java:915)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:640)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1387)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1376)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.dropTable(TestDbTxnManager2.java:974)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLockBlockedBy(TestDbTxnManager2.java:396)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
>
[jira] [Commented] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987840#comment-16987840
]
Peter Vary commented on HIVE-22580:
---
My theory is that the previous test ends, and the heartbeating thread is
interrupted inside of the derby call. MetaStoreDirectSql.ensureDbInit() catches
the InterruptedException (this happens only with EmbeddedMetaStore), and never
rethrows it. So the heartbeater never interrupted, and RetryingMetaStoreClient
retries the heartbeat. This recreates/reinitializes the txnDb, see:
{code}
2019-12-03T20:18:49,221 INFO [Heartbeater-1] txn.TxnHandler: Hacking in canned
values for transaction manager
{code}
This log line is printed just before {code}TxnDbUtil.prepDb(conf);{code} is
executed, so the heartbeater thread starts to recreate the txnDb related tables.
The main thread finishes the previous test case, and runs setup for the next
test. This also drops, and starts the txnDb related tables. But sees that the
first table is already there and so decided that the tables are already there:
{code}
2019-12-03T20:18:49,235 INFO [main] txn.TxnDbUtil: Txn tables already exist,
returning
{code}
The problem is that not all of the tables are there. So when it wants to read
the NEXT_TXN_ID it fails, since the first thread (heartbeater) is not there.
So the issue is twofold:
* Need a fix how MetaStoreDirectSql.ensureDbInit() handles InterruptedException
- this is kinda risky, might worth another jira to understand the sideeffects
of this.
* Need to synchronize TxnDbUtil.prepDb(), so no concurrent table creation
happens
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:604)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:560)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8095)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy38.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3139)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3103)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy39.openTxn(Unknown Source)
> at
>
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987838#comment-16987838
]
Vineet Garg commented on HIVE-22569:
[~belugabehr] It will be nice to leave a comment in class file to explain why
it is immutable. Otherwise +1 pending tests
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22580) Flakyness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-22580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-22580:
-
> Flakyness in TestDbTxnManager2
> --
>
> Key: HIVE-22580
> URL: https://issues.apache.org/jira/browse/HIVE-22580
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
>
> Different TestDbTxnManager2 tests are failing intermittently with:
> {code}
> 2019-12-03T20:18:49,269 DEBUG [main] pool.PoolBase: HikariPool-1 - Reset
> (autoCommit) on connection
> org.apache.derby.impl.jdbc.EmbedConnection@912054991 (XID = 347), (SESSIONID
> = 7), (DATABASE =
> memory:/data/jenkins/workspace/hive-TestDbTxnManager2/hive/ql/target/tmp/junit_metastore_db),
> (DRDAID = null)
> 2019-12-03T20:18:49,270 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to select from transaction database
> java.sql.SQLSyntaxErrorException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown
> Source)
> at
> com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:604)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:560)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:8095)
> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy38.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxnsIntr(HiveMetaStoreClient.java:3139)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxn(HiveMetaStoreClient.java:3103)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> at com.sun.proxy.$Proxy39.openTxn(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:250)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.openTxn(DbTxnManager.java:234)
> at org.apache.hadoop.hive.ql.Driver.openTransaction(Driver.java:915)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:640)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1564)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1387)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1376)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.dropTable(TestDbTxnManager2.java:974)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLockBlockedBy(TestDbTxnManager2.java:396)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
>
[jira] [Commented] (HIVE-22571) Review of ExprNodeFieldDesc Class
[ https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987823#comment-16987823 ] Vineet Garg commented on HIVE-22571: +1 pending tests > Review of ExprNodeFieldDesc Class > - > > Key: HIVE-22571 > URL: https://issues.apache.org/jira/browse/HIVE-22571 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22571.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22571) Review of ExprNodeFieldDesc Class
[
https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987802#comment-16987802
]
Hive QA commented on HIVE-22571:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987395/HIVE-22571.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19736/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19736/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987395 - PreCommit-HIVE-Build
> Review of ExprNodeFieldDesc Class
> -
>
> Key: HIVE-22571
> URL: https://issues.apache.org/jira/browse/HIVE-22571
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22571.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[
https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22579:
Description:
There is a scenario when different SplitGenerator instances try to cover the
delta-only buckets (having no base file) more than once, so there could be
multiple OrcSplit instances generated for the same delta file, causing more
tasks to read the same delta file more than once, causing duplicate records in
a simple select star query.
File structure for a 256 bucket table
{code}
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013
-rw-r--r-- 3 hive hadoop353 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00012
-rw-r--r-- 3 hive hadoop 1642 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1635 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1808 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00043
-rwxrwxrwx 3 hive hadoop 1633 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
{code}
in this case, when bucket_00171 file has a record, and there is no base file
for that, a select (*) with ETL split strategy can generate 2 splits for the
same delta bucket...
the scenario of the issue:
1. ETLSplitStrategy contains a [covered[]
array|https://github.com/apache/hive/blob/298f749ec7be04abb797fb119f3f0b923c8a1b27/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L763]
which is [shared between the SplitInfo
instances|https://github.com/apache/hive/blob/298f749ec7be04abb797fb119f3f0b923c8a1b27/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L824]
to be created
2. a SplitInfo instance is created for [every base file (2 in this
case)|https://github.com/apache/hive/blob/298f749ec7be04abb797fb119f3f0b923c8a1b27/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L809]
3. for every SplitInfo, [a SplitGenerator is
created|https://github.com/apache/hive/blob/298f749ec7be04abb797fb119f3f0b923c8a1b27/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L925-L926],
and in the constructor, [parent's getSplit is
called|https://github.com/apache/hive/blob/298f749ec7be04abb797fb119f3f0b923c8a1b27/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L1251],
which tries to take care of the deltas
I'm not sure at the moment what's the intention of this, but this way,
duplicated delta split can be generated, which can cause duplicated read later
(note that both tasks read the same delta file: bucket_00171)
{code}
2019-12-01T16:24:53,669 INFO [TezTR-127843_16_30_0_171_0
(1575040127843_0016_30_00_000171_0)] orc.ReaderImpl: Reading ORC rows from
hdfs://c3351-node2.squadron.support.hortonworks.com:8020/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
with {include: [true, true, true, true, true, true, true, true, true, true,
true, true], offset: 0, length: 9223372036854775807, schema:
struct}
2019-12-01T16:24:53,672 INFO [TezTR-127843_16_30_0_171_0
(1575040127843_0016_30_00_000171_0)] lib.MRReaderMapred: Processing split:
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat:OrcSplit
[hdfs://c3351-node2.squadron.support.hortonworks.com:8020/apps/hive/warehouse/naresh.db/test1,
start=171, length=0, isOriginal=false, fileLength=9223372036854775807,
hasFooter=false, hasBase=false, deltas=[{ minTxnId: 14 maxTxnId: 14 stmtIds:
[0] }, { minTxnId: 15 maxTxnId: 15 stmtIds: [0] }, { minTxnId: 16 maxTxnId: 16
stmtIds: [0] }]]
2019-12-01T16:24:55,807 INFO [TezTR-127843_16_30_0_425_0
(1575040127843_0016_30_00_000425_0)] orc.ReaderImpl: Reading ORC rows from
hdfs://c3351-node2.squadron.support.hortonworks.com:8020/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
with {include: [true, true, true, true, true, true, true, true, true, true,
true, true], offset: 0, length: 9223372036854775807, schema:
struct}
[jira] [Commented] (HIVE-22571) Review of ExprNodeFieldDesc Class
[
https://issues.apache.org/jira/browse/HIVE-22571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987765#comment-16987765
]
Hive QA commented on HIVE-22571:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19736/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19736/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19736/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of ExprNodeFieldDesc Class
> -
>
> Key: HIVE-22571
> URL: https://issues.apache.org/jira/browse/HIVE-22571
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22571.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22517) Sysdb related qtests also output the sysdb sql commands to q.out
[ https://issues.apache.org/jira/browse/HIVE-22517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22517: Attachment: HIVE-22517.02.patch > Sysdb related qtests also output the sysdb sql commands to q.out > > > Key: HIVE-22517 > URL: https://issues.apache.org/jira/browse/HIVE-22517 > Project: Hive > Issue Type: Improvement > Components: Test >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22517.01.patch, HIVE-22517.02.patch, > HIVE-22517.02.patch, HIVE-22517.02.patch, HIVE-22517.02.patch, > HIVE-22517.02.patch, HIVE-22517.02.patch, HIVE-22517.02.patch > > > it would be better to not have it on the outputs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987751#comment-16987751
]
Hive QA commented on HIVE-22569:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12987391/HIVE.22569.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_parts]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[column_access_stats]
(batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[column_access_stats]
(batchId=140)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testDropPartitionsWithPurge
(batchId=336)
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerCheckInvocation.testInputAllColumnsUsed
(batchId=280)
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerCheckInvocation.testPermFunction
(batchId=280)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19735/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19735/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12987391 - PreCommit-HIVE-Build
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Attachment: HIVE-22502.1.patch
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Attachment: (was: HIVE-22502.1.patch)
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22577) StringIndexOutOfBoundsException when getting sessionId from worker node name
[
https://issues.apache.org/jira/browse/HIVE-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Magyar updated HIVE-22577:
-
Attachment: HIVE-22577.1.patch
> StringIndexOutOfBoundsException when getting sessionId from worker node name
>
>
> Key: HIVE-22577
> URL: https://issues.apache.org/jira/browse/HIVE-22577
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22577.1.patch
>
>
> When the node name is "worker-" the following exception is thrown
>
> {code:java}
> Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
> range: -1Caused by: java.lang.StringIndexOutOfBoundsException: String index
> out of range: -1 at java.lang.String.substring(String.java:1931) at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.extractSeqNum(ZkRegistryBase.java:781)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase.populateCache(ZkRegistryBase.java:507)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.access$000(LlapZookeeperRegistryImpl.java:65)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl$DynamicServiceInstanceSet.(LlapZookeeperRegistryImpl.java:313)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getInstances(LlapZookeeperRegistryImpl.java:462)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapZookeeperRegistryImpl.getApplicationId(LlapZookeeperRegistryImpl.java:469)
> at
> org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService.getApplicationId(LlapRegistryService.java:212)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getCustomSplitLocationProvider(Utils.java:77)
> at
> org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:53)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:140)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[
https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22579:
Description:
There is a scenario when different SplitGenerator instances try to cover the
delta-only buckets (having no base file) more than once, so there could be
multiple OrcSplit instances generated for the same delta file, causing more
tasks to read the same delta file more than once, causing duplicate records in
a simple select star query.
File structure for a 256 bucket table
{code}
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013
-rw-r--r-- 3 hive hadoop353 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00012
-rw-r--r-- 3 hive hadoop 1642 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1635 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1808 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00043
-rwxrwxrwx 3 hive hadoop 1633 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
{code}
in this case, when bucket_00171 file has a record, and there is no base file
for that, a select (*) with ETL split strategy can generate 2 splits for the
same delta bucket...
seems like this issue doesn't affect AcidV2, as getSplits() returns an empty
collection or throws an exception in case of unexpected deltas (which was the
case here, where deltas was not unexpected):
https://github.com/apache/hive/blob/8ee3497f87f81fa84ee1023e891dc54087c2cd5e/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L1178-L1197
was:
There is a scenario when different SplitGenerator instances try to cover the
delta-only buckets (having no base file) more than once, so there could be
multiple OrcSplit instances generated for the same delta file, causing more
tasks to read the same delta file more than once, causing duplicate records in
a simple select star query.
File structure for a 256 bucket table
{code}
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013
-rw-r--r-- 3 hive hadoop353 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00012
-rw-r--r-- 3 hive hadoop 1642 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1635 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1808 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00043
-rwxrwxrwx 3 hive hadoop 1633 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
{code}
in this case, when bucket_00171 file has a record, and there is no base file
for that, a select (*) with ETL split strategy can generate 2 splits for the
same delta bucket...
> ACID v1: covered delta-only splits (without base) should be marked as covered
> (branch-2)
>
>
> Key: HIVE-22579
> URL:
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Attachment: HIVE-22408.6.patch > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Status: Open (was: Patch Available) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Status: Patch Available (was: Open) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22569) PartitionPruner use Collections Class
[
https://issues.apache.org/jira/browse/HIVE-22569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987722#comment-16987722
]
Hive QA commented on HIVE-22569:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
34s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19735/dev-support/hive-personality.sh
|
| git revision | master / 23db35e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19735/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PartitionPruner use Collections Class
> -
>
> Key: HIVE-22569
> URL: https://issues.apache.org/jira/browse/HIVE-22569
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE.22569.1.patch
>
>
> Make class {{PrunedPartitionList.java}} more immutable. It kinda is already.
> Immutable classes are easier to reason about.
> Use {{Collections.emptyXXX}} in {{PartitionPruner}} to save an object
> instantiation at runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987721#comment-16987721 ] Vineet Garg commented on HIVE-22572: [~odraese] Can you add a test for this? > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[
https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22579:
Description:
There is a scenario when different SplitGenerator instances try to cover the
delta-only buckets (having no base file) more than once, so there could be
multiple OrcSplit instances generated for the same delta file, causing more
tasks to read the same delta file more than once, causing duplicate records in
a simple select star query.
File structure for a 256 bucket table
{code}
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013
-rw-r--r-- 3 hive hadoop353 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00012
-rw-r--r-- 3 hive hadoop 1642 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/base_013/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1635 2019-11-29 15:55
/apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00012
-rwxrwxrwx 3 hive hadoop 1808 2019-11-29 16:04
/apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00140
drwxrwxrwx - hive hadoop 0 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_
-rwxrwxrwx 3 hive hadoop348 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00043
-rwxrwxrwx 3 hive hadoop 1633 2019-11-29 16:06
/apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
{code}
in this case, when bucket_00171 file has a record, and there is no base file
for that, a select (*) with ETL split strategy can generate 2 splits for the
same delta bucket...
> ACID v1: covered delta-only splits (without base) should be marked as covered
> (branch-2)
>
>
> Key: HIVE-22579
> URL: https://issues.apache.org/jira/browse/HIVE-22579
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22579.01.branch-2.patch
>
>
> There is a scenario when different SplitGenerator instances try to cover the
> delta-only buckets (having no base file) more than once, so there could be
> multiple OrcSplit instances generated for the same delta file, causing more
> tasks to read the same delta file more than once, causing duplicate records
> in a simple select star query.
> File structure for a 256 bucket table
> {code}
> drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/base_013
> -rw-r--r-- 3 hive hadoop353 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/base_013/bucket_00012
> -rw-r--r-- 3 hive hadoop 1642 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/base_013/bucket_00140
> drwxrwxrwx - hive hadoop 0 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/delta_014_014_
> -rwxrwxrwx 3 hive hadoop348 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00012
> -rwxrwxrwx 3 hive hadoop 1635 2019-11-29 15:55
> /apps/hive/warehouse/naresh.db/test1/delta_014_014_/bucket_00140
> drwxrwxrwx - hive hadoop 0 2019-11-29 16:04
> /apps/hive/warehouse/naresh.db/test1/delta_015_015_
> -rwxrwxrwx 3 hive hadoop348 2019-11-29 16:04
> /apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00012
> -rwxrwxrwx 3 hive hadoop 1808 2019-11-29 16:04
> /apps/hive/warehouse/naresh.db/test1/delta_015_015_/bucket_00140
> drwxrwxrwx - hive hadoop 0 2019-11-29 16:06
> /apps/hive/warehouse/naresh.db/test1/delta_016_016_
> -rwxrwxrwx 3 hive hadoop348 2019-11-29 16:06
> /apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00043
> -rwxrwxrwx 3 hive hadoop 1633 2019-11-29 16:06
> /apps/hive/warehouse/naresh.db/test1/delta_016_016_/bucket_00171
> {code}
> in this case, when bucket_00171 file has a record, and there is no base file
> for that, a select (*) with ETL split strategy can generate 2 splits for the
> same delta bucket...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22579: Status: Patch Available (was: Open) > ACID v1: covered delta-only splits (without base) should be marked as covered > (branch-2) > > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-22579.01.branch-2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22489) Reduce Sink operator should order nulls by parameter
[
https://issues.apache.org/jira/browse/HIVE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22489:
--
Status: Patch Available (was: Open)
> Reduce Sink operator should order nulls by parameter
> -
>
> Key: HIVE-22489
> URL: https://issues.apache.org/jira/browse/HIVE-22489
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22489.1.patch, HIVE-22489.2.patch,
> HIVE-22489.3.patch, HIVE-22489.3.patch
>
>
> When the property hive.default.nulls.last is set to true and no null order is
> explicitly specified in the ORDER BY clause of the query null ordering should
> be NULLS LAST.
> But some of the Reduce Sink operators still orders null first.
> {code}
> SET hive.default.nulls.last=true;
> EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key LIMIT 5;
> {code}
> {code}
> PREHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> A masked pattern was here
> POSTHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@src
> A masked pattern was here
> OPTIMIZED SQL: SELECT `t0`.`key`, `t2`.`value`
> FROM (SELECT `key`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t0`
> INNER JOIN (SELECT `key`, `value`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
> ORDER BY `t0`.`key`
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: src1
> filterExpr: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Filter Operator
> isSamplingPred: false
> predicate: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> null sort order: a
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 500 Data size: 43500 Basic
> stats: COMPLETE Column stats: COMPLETE
> tag: 0
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: src
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns key,value
> columns.comments 'default','default'
> columns.types string:string
> A masked pattern was here
> name default.src
> numFiles 1
> numRows 500
> rawDataSize 5312
> serialization.ddl struct src { string key, string value}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 5812
> A masked pattern was here
> serde:
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22579: Attachment: HIVE-22579.01.branch-2.patch > ACID v1: covered delta-only splits (without base) should be marked as covered > (branch-2) > > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-22579.01.branch-2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22489) Reduce Sink operator should order nulls by parameter
[
https://issues.apache.org/jira/browse/HIVE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22489:
--
Status: Open (was: Patch Available)
> Reduce Sink operator should order nulls by parameter
> -
>
> Key: HIVE-22489
> URL: https://issues.apache.org/jira/browse/HIVE-22489
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22489.1.patch, HIVE-22489.2.patch,
> HIVE-22489.3.patch, HIVE-22489.3.patch
>
>
> When the property hive.default.nulls.last is set to true and no null order is
> explicitly specified in the ORDER BY clause of the query null ordering should
> be NULLS LAST.
> But some of the Reduce Sink operators still orders null first.
> {code}
> SET hive.default.nulls.last=true;
> EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key LIMIT 5;
> {code}
> {code}
> PREHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> A masked pattern was here
> POSTHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@src
> A masked pattern was here
> OPTIMIZED SQL: SELECT `t0`.`key`, `t2`.`value`
> FROM (SELECT `key`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t0`
> INNER JOIN (SELECT `key`, `value`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
> ORDER BY `t0`.`key`
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: src1
> filterExpr: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Filter Operator
> isSamplingPred: false
> predicate: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> null sort order: a
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 500 Data size: 43500 Basic
> stats: COMPLETE Column stats: COMPLETE
> tag: 0
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: src
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns key,value
> columns.comments 'default','default'
> columns.types string:string
> A masked pattern was here
> name default.src
> numFiles 1
> numRows 500
> rawDataSize 5312
> serialization.ddl struct src { string key, string value}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 5812
> A masked pattern was here
> serde:
[jira] [Updated] (HIVE-22489) Reduce Sink operator should order nulls by parameter
[
https://issues.apache.org/jira/browse/HIVE-22489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22489:
--
Attachment: HIVE-22489.3.patch
> Reduce Sink operator should order nulls by parameter
> -
>
> Key: HIVE-22489
> URL: https://issues.apache.org/jira/browse/HIVE-22489
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22489.1.patch, HIVE-22489.2.patch,
> HIVE-22489.3.patch, HIVE-22489.3.patch
>
>
> When the property hive.default.nulls.last is set to true and no null order is
> explicitly specified in the ORDER BY clause of the query null ordering should
> be NULLS LAST.
> But some of the Reduce Sink operators still orders null first.
> {code}
> SET hive.default.nulls.last=true;
> EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key LIMIT 5;
> {code}
> {code}
> PREHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> A masked pattern was here
> POSTHOOK: query: EXPLAIN EXTENDED
> SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
> src2.key) ORDER BY src1.key
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@src
> A masked pattern was here
> OPTIMIZED SQL: SELECT `t0`.`key`, `t2`.`value`
> FROM (SELECT `key`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t0`
> INNER JOIN (SELECT `key`, `value`
> FROM `default`.`src`
> WHERE `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
> ORDER BY `t0`.`key`
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: src1
> filterExpr: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Filter Operator
> isSamplingPred: false
> predicate: key is not null (type: boolean)
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 43500 Basic stats:
> COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> null sort order: a
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 500 Data size: 43500 Basic
> stats: COMPLETE Column stats: COMPLETE
> tag: 0
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: src
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns key,value
> columns.comments 'default','default'
> columns.types string:string
> A masked pattern was here
> name default.src
> numFiles 1
> numRows 500
> rawDataSize 5312
> serialization.ddl struct src { string key, string value}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 5812
> A masked pattern was here
> serde:
[jira] [Updated] (HIVE-22579) ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22579: Summary: ACID v1: covered delta-only splits (without base) should be marked as covered (branch-2) (was: ACID v1: covered delta splits (without base) should be marked as covered (branch-2)) > ACID v1: covered delta-only splits (without base) should be marked as covered > (branch-2) > > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta splits (without base) should be marked as covered (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22579: Summary: ACID v1: covered delta splits (without base) should be marked as covered (branch-2) (was: ACID v1: covered delta splits (without base) should be marked as covered) > ACID v1: covered delta splits (without base) should be marked as covered > (branch-2) > --- > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22579) ACID v1: covered delta splits (without base) should be marked as covered
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22579: Summary: ACID v1: covered delta splits (without base) should be marked as covered (was: ACID: covered delta splits (without base) should be marked as covered) > ACID v1: covered delta splits (without base) should be marked as covered > > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22579) ACID: covered delta splits (without base) should be marked as covered
[ https://issues.apache.org/jira/browse/HIVE-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-22579: --- Assignee: László Bodor > ACID: covered delta splits (without base) should be marked as covered > - > > Key: HIVE-22579 > URL: https://issues.apache.org/jira/browse/HIVE-22579 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[ https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aron Hamvas reassigned HIVE-22578: -- > CBO genOPTree is not failsafe for CTAS and VIEW statements > -- > > Key: HIVE-22578 > URL: https://issues.apache.org/jira/browse/HIVE-22578 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > > If CBO fails during OT generation, it is supposed to skip CBO and fall back > to SemanticAnalyzer generating the OT for the original AST. However, for CTAS > and VIEW statements, the original AST is discarded and replaced with the new > AST. > If OT generation for the new AST fails with a SemanticException, the > compilation is destined to fail completely since we cannot fall back to the > original and possibly valid AST. > Also, the SemanticException sometimes gets hidden and misjudged as a missing > column statistics issue due to bad error handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
