[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754574#comment-16754574
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956626/HIVE-21052.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15728 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[1]
(batchId=209)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[2]
(batchId=209)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15818/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15818/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15818/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956626 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754555#comment-16754555
]
Hive QA commented on HIVE-21052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
15s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
12s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
33s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/common: The patch generated 1 new + 95 unchanged
- 0 fixed = 96 total (was 95) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/0.23: The patch generated 5 new + 69 unchanged -
0 fixed = 74 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 28 new + 571 unchanged - 11
fixed = 599 total (was 582) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 149
unchanged - 0 fixed = 159 total (was 149) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 3 new
+ 184 unchanged - 0 fixed = 187 total (was 184) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
43s{color} | {color:red} ql generated 2 new + 2303 unchanged - 1 fixed = 2305
total (was 2304) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean()
may fail to close PreparedStatement At
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[
https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21029:
Description:
Existing deployments using hive replication do not get external tables
replicated. For such deployments to enable external table replication they will
have to provide a specific switch to first bootstrap external tables as part of
hive incremental replication, following which the incremental replication will
take care of further changes in external tables.
The switch will be provided by an additional hive configuration (for ex:
hive.repl.bootstrap.external.tables) and is to be used in
{code} WITH {code} clause of
{code} REPL DUMP {code} command.
Additionally the existing hive config _hive.repl.include.external.tables_ will
always have to be set to "true" in the above clause.
Proposed usage for enabling external tables replication on existing replication
policy.
1. Consider an ongoing repl policy in incremental phase.
Enable hive.repl.include.external.tables=true and
hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP.
- Dumps all events but skips events related to external tables.
- Instead, combine bootstrap dump for all external tables under “_bootstrap”
directory.
- Also, includes the data locations file "_external_tables_info”.
- LIMIT or TO clause shouldn’t be there to ensure the latest events are dumped
before bootstrap dumping external tables.
2. REPL LOAD on this dump applies all the events first, copies external tables
data and then bootstrap external tables (metadata).
- It is possible that the external tables (metadata) are not point-in time
consistent with rest of the tables.
- But, it would be eventually consistent when the next incremental load is
applied.
- This REPL LOAD is fault tolerant and can be retried if failed.
3. All future REPL DUMPs on this repl policy should set
hive.repl.bootstrap.external.tables=false.
- If not set to false, then target might end up having inconsistent set of
external tables as bootstrap wouldn’t clean-up any dropped external tables.
was:
Existing deployments using hive replication do not get external tables
replicated. For such deployments to enable external table replication they will
have to provide a specific switch to first bootstrap external tables as part of
hive incremental replication, following which the incremental replication will
take care of further changes in external tables.
The switch will be provided by an additional hive configuration (for ex:
hive.repl.bootstrap.external.tables) and is to be used in
{code} WITH {code} clause of
{code} REPL DUMP {code} command.
Additionally the existing hive config _hive.repl.include.external.tables_ will
always have to be set to "true" in the above clause.
Proposed usage for enabling external tables replication on existing DLM
replication policy.
1. Consider an ongoing repl policy in incremental phase.
Enable hive.repl.include.external.tables=true and
hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP.
- Dumps all events but skips events related to external tables.
- Instead, combine bootstrap dump for all external tables under “_bootstrap”
directory.
- Also, includes the data locations file "_external_tables_info”.
- LIMIT or TO clause shouldn’t be there to ensure the latest events are dumped
before bootstrap dumping external tables.
2. REPL LOAD on this dump applies all the events first, copies external tables
data and then bootstrap external tables (metadata).
- It is possible that the external tables (metadata) are not point-in time
consistent with rest of the tables.
- But, it would be eventually consistent when the next incremental load is
applied.
- This REPL LOAD is fault tolerant and can be retried if failed.
3. All future REPL DUMPs on this repl policy should set
hive.repl.bootstrap.external.tables=false.
- If not set to false, then target might end up having inconsistent set of
external tables as bootstrap wouldn’t clean-up any dropped external tables.
> External table replication for existing deployments running incremental
> replication.
>
>
> Key: HIVE-21029
> URL: https://issues.apache.org/jira/browse/HIVE-21029
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 3.1.1
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Critical
> Fix For: 4.0.0
>
>
> Existing deployments using hive replication do not get external tables
> replicated. For such deployments to enable external table replication they
> will have to provide a specific switch to first bootstrap external tables as
> part of hive incremental replication, following which the
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[
https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21029:
Description:
Existing deployments using hive replication do not get external tables
replicated. For such deployments to enable external table replication they will
have to provide a specific switch to first bootstrap external tables as part of
hive incremental replication, following which the incremental replication will
take care of further changes in external tables.
The switch will be provided by an additional hive configuration (for ex:
hive.repl.bootstrap.external.tables) and is to be used in
{code} WITH {code} clause of
{code} REPL DUMP {code} command.
Additionally the existing hive config _hive.repl.include.external.tables_ will
always have to be set to "true" in the above clause.
Proposed usage for enabling external tables replication on existing DLM
replication policy.
1. Consider an ongoing repl policy in incremental phase.
Enable hive.repl.include.external.tables=true and
hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP.
- Dumps all events but skips events related to external tables.
- Instead, combine bootstrap dump for all external tables under “_bootstrap”
directory.
- Also, includes the data locations file "_external_tables_info”.
- LIMIT or TO clause shouldn’t be there to ensure the latest events are dumped
before bootstrap dumping external tables.
2. REPL LOAD on this dump applies all the events first, copies external tables
data and then bootstrap external tables (metadata).
- It is possible that the external tables (metadata) are not point-in time
consistent with rest of the tables.
- But, it would be eventually consistent when the next incremental load is
applied.
- This REPL LOAD is fault tolerant and can be retried if failed.
3. All future REPL DUMPs on this repl policy should set
hive.repl.bootstrap.external.tables=false.
- If not set to false, then target might end up having inconsistent set of
external tables as bootstrap wouldn’t clean-up any dropped external tables.
was:
Existing deployments using hive replication do not get external tables
replicated. For such deployments to enable external table replication they will
have to provide a specific switch to first bootstrap external tables as part of
hive incremental replication, following which the incremental replication will
take care of further changes in external tables.
The switch will be provided by an additional hive configuration (for ex:
hive.repl.bootstrap.external.tables) and is to be used in
{code} WITH {code} clause of
{code} REPL DUMP {code} command.
Additionally the existing hive config _hive.repl.include.external.tables_ will
always have to be set to "true" in the above clause.
> External table replication for existing deployments running incremental
> replication.
>
>
> Key: HIVE-21029
> URL: https://issues.apache.org/jira/browse/HIVE-21029
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 3.1.1
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Critical
> Fix For: 4.0.0
>
>
> Existing deployments using hive replication do not get external tables
> replicated. For such deployments to enable external table replication they
> will have to provide a specific switch to first bootstrap external tables as
> part of hive incremental replication, following which the incremental
> replication will take care of further changes in external tables.
> The switch will be provided by an additional hive configuration (for ex:
> hive.repl.bootstrap.external.tables) and is to be used in
> {code} WITH {code} clause of
> {code} REPL DUMP {code} command.
> Additionally the existing hive config _hive.repl.include.external.tables_
> will always have to be set to "true" in the above clause.
> Proposed usage for enabling external tables replication on existing DLM
> replication policy.
> 1. Consider an ongoing repl policy in incremental phase.
> Enable hive.repl.include.external.tables=true and
> hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP.
> - Dumps all events but skips events related to external tables.
> - Instead, combine bootstrap dump for all external tables under “_bootstrap”
> directory.
> - Also, includes the data locations file "_external_tables_info”.
> - LIMIT or TO clause shouldn’t be there to ensure the latest events are
> dumped before bootstrap dumping external tables.
> 2. REPL LOAD on this dump applies all the events first, copies external
> tables data and then bootstrap external tables (metadata).
> - It is possible that the external tables (metadata) are not
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~ashutoshc] for taking a look.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754525#comment-16754525
]
Hive QA commented on HIVE-17020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956625/HIVE-17020.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15720 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15817/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15817/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15817/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956625 - PreCommit-HIVE-Build
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754507#comment-16754507
]
Eugene Koifman commented on HIVE-20699:
---
I left a few of minor comments on RB for patch 9. Overall LGTM, except for
{{HIVE_TRANSACTIONAL_TABLE_SCAN}} not being visible in {{HiveSplitGenerator}}.
While not caused by this patch I don't think it can properly work in production
w/o it being fixed.
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch,
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch,
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754497#comment-16754497
]
Hive QA commented on HIVE-17020:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} ql: The patch generated 0 new + 28 unchanged - 1
fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15817/dev-support/hive-personality.sh
|
| git revision | master / 698206a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15817/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21128) hive.version.shortname should be 3.2 on branch-3
[
https://issues.apache.org/jira/browse/HIVE-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754483#comment-16754483
]
Hive QA commented on HIVE-21128:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956620/HIVE-21128.03.branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 113 failed/errored test(s), 14481 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=274)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=274)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=239)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestGetPartitionsUsingProjectionAndFilterSpecs - did not produce a TEST-*.xml
file (likely timed out) (batchId=230)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=235)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=233)
TestHiveMetaToolCommandLine - did not produce a TEST-*.xml file (likely timed
out) (batchId=235)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=235)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestMarkPartition - did not produce a
[jira] [Commented] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[
https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754460#comment-16754460
]
Hive QA commented on HIVE-21175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} llap-common in master has 76 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} spark-client in master has 10 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} llap-server in master has 81 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} beeline in master has 53 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hplsql in master has 176 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings.
[jira] [Commented] (HIVE-21128) hive.version.shortname should be 3.2 on branch-3
[
https://issues.apache.org/jira/browse/HIVE-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754455#comment-16754455
]
Hive QA commented on HIVE-21128:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 23s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-15816/patches/PreCommit-HIVE-Build-15816.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15816/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive.version.shortname should be 3.2 on branch-3
>
>
> Key: HIVE-21128
> URL: https://issues.apache.org/jira/browse/HIVE-21128
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21128.01.branch-3.patch,
> HIVE-21128.02.branch-3.patch, HIVE-21128.03.branch-3.patch
>
>
> Since 3.1.0 is already release, the {{hive.version.shortname}} property in
> the pom.xml of standalone-metastore should be 3.2.0. This version shortname
> is used to generate the metastore schema version and used by Schematool to
> initialize the schema using the correct script. Currently it using 3.1.0
> schema init script instead of 3.2.0 init script
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[ https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754454#comment-16754454 ] BELUGA BEHR commented on HIVE-21175: [~bslim] You were so helpful with [HIVE-21148]... would you mind taking a look at this too? :) > Use StandardCharsets Where Possible (Part 2) > > > Key: HIVE-21175 > URL: https://issues.apache.org/jira/browse/HIVE-21175 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21175.1.patch > > > Additional work not already addressed by [HIVE-21148]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[
https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754452#comment-16754452
]
Hive QA commented on HIVE-21175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956617/HIVE-21175.1.patch
{color:green}SUCCESS:{color} +1 due to 35 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15720 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15815/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15815/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15815/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956617 - PreCommit-HIVE-Build
> Use StandardCharsets Where Possible (Part 2)
>
>
> Key: HIVE-21175
> URL: https://issues.apache.org/jira/browse/HIVE-21175
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21175.1.patch
>
>
> Additional work not already addressed by [HIVE-21148].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21159) Modify Merge statement logic to perform Update split early
[
https://issues.apache.org/jira/browse/HIVE-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754419#comment-16754419
]
Hive QA commented on HIVE-21159:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956614/HIVE-21159.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15718 tests
executed
*Failed tests:*
{noformat}
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=251)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15814/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15814/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15814/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956614 - PreCommit-HIVE-Build
> Modify Merge statement logic to perform Update split early
> --
>
> Key: HIVE-21159
> URL: https://issues.apache.org/jira/browse/HIVE-21159
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-21159.01.patch, HIVE-21159.02.patch,
> HIVE-21159.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754416#comment-16754416 ] Jaume M commented on HIVE-21052: [~ekoifman] I've added a CleanerExecutorService to come around this problem. It only submits the task if it can acquire the lock. I've updated the rb. > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0, 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.12.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1, 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1, 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Open (was: Patch Available)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Patch Available (was: Open)
Uploading rebased patch for a final test run
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Attachment: HIVE-17020.10.patch
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.10.patch,
> HIVE-17020.2.patch, HIVE-17020.3.patch, HIVE-17020.4.patch,
> HIVE-17020.5.patch, HIVE-17020.6.patch, HIVE-17020.7.patch,
> HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21168) Fix TestSchemaToolCatalogOps
[
https://issues.apache.org/jira/browse/HIVE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754402#comment-16754402
]
Vihang Karajgaonkar commented on HIVE-21168:
Looked into this and its related to HIVE-21128. Since the hiveversion is still
3.1 in the pom.xml, schemaTool is using 3.1 schema init scripts which do not
have the {{CREATE_TIME}} column added in HIVE-21077
> Fix TestSchemaToolCatalogOps
>
>
> Key: HIVE-21168
> URL: https://issues.apache.org/jira/browse/HIVE-21168
> Project: Hive
> Issue Type: Test
>Affects Versions: 3.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> HIVE-21077 causes TestSchemaToolCatalogOps to fail on branch-3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21128) hive.version.shortname should be 3.2 on branch-3
[
https://issues.apache.org/jira/browse/HIVE-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21128:
---
Attachment: HIVE-21128.03.branch-3.patch
> hive.version.shortname should be 3.2 on branch-3
>
>
> Key: HIVE-21128
> URL: https://issues.apache.org/jira/browse/HIVE-21128
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21128.01.branch-3.patch,
> HIVE-21128.02.branch-3.patch, HIVE-21128.03.branch-3.patch
>
>
> Since 3.1.0 is already release, the {{hive.version.shortname}} property in
> the pom.xml of standalone-metastore should be 3.2.0. This version shortname
> is used to generate the metastore schema version and used by Schematool to
> initialize the schema using the correct script. Currently it using 3.1.0
> schema init script instead of 3.2.0 init script
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21159) Modify Merge statement logic to perform Update split early
[
https://issues.apache.org/jira/browse/HIVE-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754401#comment-16754401
]
Hive QA commented on HIVE-21159:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 15 new + 1787 unchanged - 26
fixed = 1802 total (was 1813) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15814/dev-support/hive-personality.sh
|
| git revision | master / 698206a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15814/yetus/diff-checkstyle-ql.txt
|
| modules | C: common standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15814/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Modify Merge statement logic to perform Update split early
> --
>
> Key: HIVE-21159
> URL: https://issues.apache.org/jira/browse/HIVE-21159
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-21159.01.patch, HIVE-21159.02.patch,
> HIVE-21159.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[ https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HIVE-21175: -- Assignee: BELUGA BEHR > Use StandardCharsets Where Possible (Part 2) > > > Key: HIVE-21175 > URL: https://issues.apache.org/jira/browse/HIVE-21175 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21175.1.patch > > > Additional work not already addressed by [HIVE-21148]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754369#comment-16754369
]
Hive QA commented on HIVE-20699:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956608/HIVE-20699.9.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15725 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15813/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15813/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15813/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956608 - PreCommit-HIVE-Build
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch,
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch,
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[ https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21175: --- Status: Patch Available (was: Open) > Use StandardCharsets Where Possible (Part 2) > > > Key: HIVE-21175 > URL: https://issues.apache.org/jira/browse/HIVE-21175 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21175.1.patch > > > Additional work not already addressed by [HIVE-21148]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21175) Use StandardCharsets Where Possible (Part 2)
[ https://issues.apache.org/jira/browse/HIVE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21175: --- Attachment: HIVE-21175.1.patch > Use StandardCharsets Where Possible (Part 2) > > > Key: HIVE-21175 > URL: https://issues.apache.org/jira/browse/HIVE-21175 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21175.1.patch > > > Additional work not already addressed by [HIVE-21148]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21148) Use StandardCharsets Where Possible
[
https://issues.apache.org/jira/browse/HIVE-21148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21148:
---
Summary: Use StandardCharsets Where Possible (was: Remove Use
StandardCharsets Where Possible)
> Use StandardCharsets Where Possible
> ---
>
> Key: HIVE-21148
> URL: https://issues.apache.org/jira/browse/HIVE-21148
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21148.1.patch
>
>
> Starting in Java 1.7, JDKs must support a set of standard charsets. When
> using this facility, instead of passing around the name (string) of the
> character set, one can pass around a {{Charset}} object and there is no need
> to catch a {{UnsupportedEncodingException}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21159) Modify Merge statement logic to perform Update split early
[ https://issues.apache.org/jira/browse/HIVE-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754349#comment-16754349 ] Eugene Koifman commented on HIVE-21159: --- fixed test failures in patch 3. [~vgumashta] could you review please > Modify Merge statement logic to perform Update split early > -- > > Key: HIVE-21159 > URL: https://issues.apache.org/jira/browse/HIVE-21159 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21159.01.patch, HIVE-21159.02.patch, > HIVE-21159.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21159) Modify Merge statement logic to perform Update split early
[ https://issues.apache.org/jira/browse/HIVE-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21159: -- Attachment: HIVE-21159.03.patch > Modify Merge statement logic to perform Update split early > -- > > Key: HIVE-21159 > URL: https://issues.apache.org/jira/browse/HIVE-21159 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21159.01.patch, HIVE-21159.02.patch, > HIVE-21159.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754339#comment-16754339
]
Hive QA commented on HIVE-20699:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 19 new + 520 unchanged - 1
fixed = 539 total (was 521) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 184
unchanged - 0 fixed = 205 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 25 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
48s{color} | {color:red} ql generated 2 new + 2304 unchanged - 0 fixed = 2306
total (was 2304) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
52s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
Table, Partition, String) concatenates strings using + in a loop At
CompactorMR.java:strings using + in a loop At CompactorMR.java:[line 534] |
| |
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses
Object.equals() At GenericUDFValidateAcidSortOrder.java:Object.equals() At
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15813/dev-support/hive-personality.sh
|
| git revision | master / 698206a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15813/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15813/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754317#comment-16754317
]
Hive QA commented on HIVE-20255:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956599/HIVE-20255.14.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15717 tests
executed
*Failed tests:*
{noformat}
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=251)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=190)
org.apache.hive.minikdc.TestSSLWithMiniKdc.testConnection (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15812/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15812/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15812/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956599 - PreCommit-HIVE-Build
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-20699:
Attachment: HIVE-20699.9.patch
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch,
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch,
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21144) ODBC with prepared statement fail inside a CTE
[
https://issues.apache.org/jira/browse/HIVE-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754288#comment-16754288
]
Shawn Weeks commented on HIVE-21144:
I suspect it's going to be an issue with the Hortonworks ODBC Driver which is
licensed from Simba. I don't believe Hive itself actually supports binding
parameters so everything is being handled by the driver. I know on the JDBC
side even when you use parameters the stringified version is what Hive sees
with the parameters expanded.
> ODBC with prepared statement fail inside a CTE
> --
>
> Key: HIVE-21144
> URL: https://issues.apache.org/jira/browse/HIVE-21144
> Project: Hive
> Issue Type: Bug
> Components: ODBC
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> I am trying to execute a very simple query, using python/pyodbc on Windows
> (with a working system-wide odbc DSN: HiveProd):
> {code:java}
> import pyodbc cnxn = pyodbc.connect('DSN=HiveProd', autocommit=True)
> cursor = cnxn.cursor()
> # works
> q="select ? as lic, ? as cpg"
> # fails
> q="with init as (select ? as lic, ? as cpg) select * from init"
> cursor.execute(q, '1', 'some string')
> for row in cursor:
> print(row.lic, row.cpg)
> {code}
> Basically, create an odbc connection, run a query with a prepared statement
> and print the result.
> A basic query works fine. If I put this query inside a CTE, I get:
> {{ cursor.execute("with init as (select ? as lic, ? as cpg) select * from
> init", '1', 'some string') pyodbc.ProgrammingError: ('42000', "[42000]
> [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server
> while executing query. Error message from server: Error while compiling
> statement: FAILED: ParseException line 1:21 can not recognize input near '?'
> 'as' 'lic' in select clause (80) (SQLPrepare)")}}
> This is not specific to python as I get the same issue with .Net.
> Trying the same with JDBC works fine.
> Testing on Hive 3.1.0 from Hdp 3.1.0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754283#comment-16754283
]
Ashutosh Chauhan commented on HIVE-17020:
-
+1
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch,
> HIVE-17020.6.patch, HIVE-17020.7.patch, HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754273#comment-16754273
]
Vineet Garg commented on HIVE-17020:
[~ashutoshc] Can you take a look please?
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch,
> HIVE-17020.6.patch, HIVE-17020.7.patch, HIVE-17020.8.patch, HIVE-17020.9.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754259#comment-16754259
]
Vineet Garg commented on HIVE-21132:
Pushed to master, thanks [~jdere]
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch, HIVE-21132.2.patch,
> HIVE-21132.3.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate,
[jira] [Updated] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21132:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21132.1.patch, HIVE-21132.2.patch,
> HIVE-21132.3.patch, HIVE-21132.4.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and
[jira] [Updated] (HIVE-21083) Remove the requirement to specify the truststore location when TLS to the database is turned on
[
https://issues.apache.org/jira/browse/HIVE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21083:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> Remove the requirement to specify the truststore location when TLS to the
> database is turned on
> ---
>
> Key: HIVE-21083
> URL: https://issues.apache.org/jira/browse/HIVE-21083
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21083.2.patch, HIVE-21083.4.patch, HIVE-21083.patch
>
>
> In the current implementation,
> [ObjectStore.configureSSL|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L349-L382]
> throws an exception if TLS to the database is turned on
> (_metastore.dbaccess.ssl.use.SSL_) but a truststore file location
> (_metastore.dbaccess.ssl.truststore.path_) is not specified.
> However, according to the [JSSE (Java 8)
> documentation|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization],
> the Java truststore file location system property
> (_javax.net.ssl.trustStore_) defaults to using the "_jssecacerts_, if it
> exists. Otherwise, _cacerts_" files. These are the default truststores that
> come with the Java installation and contain a list of well-known certificate
> authorities.
> It was identified that one valid way of configuring TLS is by adding to these
> default files. In that case, no changes to the truststore properties are
> necessary. We should support this case by changing the following logic to
> remove the requirement for the truststore file location config property:
> {code:java}
> String trustStorePath = MetastoreConf.getVar(conf,
> ConfVars.DBACCESS_SSL_TRUSTSTORE_PATH).trim();
> if (trustStorePath.isEmpty()) {
> throw new IllegalArgumentException("SSL to the database store has
> been enabled but " +
> ConfVars.DBACCESS_SSL_TRUSTSTORE_PATH.toString() + " is empty. "
> + "Set this property to enable SSL.");
> }
> {code}
> We should also loosen the requirement on the truststore password if the user
> decides to use the Java defaults
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21083) Remove the requirement to specify the truststore location when TLS to the database is turned on
[
https://issues.apache.org/jira/browse/HIVE-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754268#comment-16754268
]
Vihang Karajgaonkar commented on HIVE-21083:
Patch merged into master. Thanks for your contribution [~mramdenbourg]
> Remove the requirement to specify the truststore location when TLS to the
> database is turned on
> ---
>
> Key: HIVE-21083
> URL: https://issues.apache.org/jira/browse/HIVE-21083
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
> Attachments: HIVE-21083.2.patch, HIVE-21083.4.patch, HIVE-21083.patch
>
>
> In the current implementation,
> [ObjectStore.configureSSL|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L349-L382]
> throws an exception if TLS to the database is turned on
> (_metastore.dbaccess.ssl.use.SSL_) but a truststore file location
> (_metastore.dbaccess.ssl.truststore.path_) is not specified.
> However, according to the [JSSE (Java 8)
> documentation|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization],
> the Java truststore file location system property
> (_javax.net.ssl.trustStore_) defaults to using the "_jssecacerts_, if it
> exists. Otherwise, _cacerts_" files. These are the default truststores that
> come with the Java installation and contain a list of well-known certificate
> authorities.
> It was identified that one valid way of configuring TLS is by adding to these
> default files. In that case, no changes to the truststore properties are
> necessary. We should support this case by changing the following logic to
> remove the requirement for the truststore file location config property:
> {code:java}
> String trustStorePath = MetastoreConf.getVar(conf,
> ConfVars.DBACCESS_SSL_TRUSTSTORE_PATH).trim();
> if (trustStorePath.isEmpty()) {
> throw new IllegalArgumentException("SSL to the database store has
> been enabled but " +
> ConfVars.DBACCESS_SSL_TRUSTSTORE_PATH.toString() + " is empty. "
> + "Set this property to enable SSL.");
> }
> {code}
> We should also loosen the requirement on the truststore password if the user
> decides to use the Java defaults
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21144) ODBC with prepared statement fail inside a CTE
[
https://issues.apache.org/jira/browse/HIVE-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754266#comment-16754266
]
Guillaume commented on HIVE-21144:
--
[~Absolutesantaja] I can confirm that I am using "native queries" I have been
bitten enough by the rewrites in the past.
In the hiveserver2 logs, I see exactly the query as I typed it, with the full
stack trace of the ParseException, so nothing new, unfortunately.
> ODBC with prepared statement fail inside a CTE
> --
>
> Key: HIVE-21144
> URL: https://issues.apache.org/jira/browse/HIVE-21144
> Project: Hive
> Issue Type: Bug
> Components: ODBC
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> I am trying to execute a very simple query, using python/pyodbc on Windows
> (with a working system-wide odbc DSN: HiveProd):
> {code:java}
> import pyodbc cnxn = pyodbc.connect('DSN=HiveProd', autocommit=True)
> cursor = cnxn.cursor()
> # works
> q="select ? as lic, ? as cpg"
> # fails
> q="with init as (select ? as lic, ? as cpg) select * from init"
> cursor.execute(q, '1', 'some string')
> for row in cursor:
> print(row.lic, row.cpg)
> {code}
> Basically, create an odbc connection, run a query with a prepared statement
> and print the result.
> A basic query works fine. If I put this query inside a CTE, I get:
> {{ cursor.execute("with init as (select ? as lic, ? as cpg) select * from
> init", '1', 'some string') pyodbc.ProgrammingError: ('42000', "[42000]
> [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server
> while executing query. Error message from server: Error while compiling
> statement: FAILED: ParseException line 1:21 can not recognize input near '?'
> 'as' 'lic' in select clause (80) (SQLPrepare)")}}
> This is not specific to python as I get the same issue with .Net.
> Trying the same with JDBC works fine.
> Testing on Hive 3.1.0 from Hdp 3.1.0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754265#comment-16754265
]
Hive QA commented on HIVE-20255:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 2
fixed = 1 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15812/dev-support/hive-personality.sh
|
| git revision | master / fd92d88 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15812/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21132:
---
Attachment: HIVE-21132.4.patch
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch, HIVE-21132.2.patch,
> HIVE-21132.3.patch, HIVE-21132.4.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by
[jira] [Commented] (HIVE-21144) ODBC with prepared statement fail inside a CTE
[
https://issues.apache.org/jira/browse/HIVE-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754256#comment-16754256
]
Shawn Weeks commented on HIVE-21144:
Check and make sure that the ODBC Driver your using is using "Native Queries".
Several of the Hive ODBC Drivers try and rewrite your queries and may be
messing with the CTE. Also if possible check the hiverserver2 logs and see if
you can figure out exactly what query Hive received.
> ODBC with prepared statement fail inside a CTE
> --
>
> Key: HIVE-21144
> URL: https://issues.apache.org/jira/browse/HIVE-21144
> Project: Hive
> Issue Type: Bug
> Components: ODBC
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> I am trying to execute a very simple query, using python/pyodbc on Windows
> (with a working system-wide odbc DSN: HiveProd):
> {code:java}
> import pyodbc cnxn = pyodbc.connect('DSN=HiveProd', autocommit=True)
> cursor = cnxn.cursor()
> # works
> q="select ? as lic, ? as cpg"
> # fails
> q="with init as (select ? as lic, ? as cpg) select * from init"
> cursor.execute(q, '1', 'some string')
> for row in cursor:
> print(row.lic, row.cpg)
> {code}
> Basically, create an odbc connection, run a query with a prepared statement
> and print the result.
> A basic query works fine. If I put this query inside a CTE, I get:
> {{ cursor.execute("with init as (select ? as lic, ? as cpg) select * from
> init", '1', 'some string') pyodbc.ProgrammingError: ('42000', "[42000]
> [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server
> while executing query. Error message from server: Error while compiling
> statement: FAILED: ParseException line 1:21 can not recognize input near '?'
> 'as' 'lic' in select clause (80) (SQLPrepare)")}}
> This is not specific to python as I get the same issue with .Net.
> Trying the same with JDBC works fine.
> Testing on Hive 3.1.0 from Hdp 3.1.0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.7.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Status: Patch Available (was: Open)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754239#comment-16754239
]
Jason Dere commented on HIVE-21132:
---
+1
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch, HIVE-21132.2.patch,
> HIVE-21132.3.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by o_totalprice
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Status: Open (was: Patch Available)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: HIVE-20255.14.patch
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.8.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.6.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.1.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.2.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.5.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.3.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: (was: HIVE-20255.4.patch)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch,
> HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.8.patch,
> HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754231#comment-16754231
]
Hive QA commented on HIVE-20255:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956589/HIVE-20255.13.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15719 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
(batchId=190)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[0]
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15811/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956589 - PreCommit-HIVE-Build
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.10.patch,
> HIVE-20255.11.patch, HIVE-20255.12.patch, HIVE-20255.13.patch,
> HIVE-20255.2.patch, HIVE-20255.3.patch, HIVE-20255.4.patch,
> HIVE-20255.5.patch, HIVE-20255.6.patch, HIVE-20255.7.patch,
> HIVE-20255.8.patch, HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue
[ https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Junge reassigned HIVE-21174: -- > hive.stats.ndv.error parameter documentation issue > -- > > Key: HIVE-21174 > URL: https://issues.apache.org/jira/browse/HIVE-21174 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, > 2.1.1, 2.1.0, 2.0.1, 2.0.0, 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, > 3.2.0, 3.1.2 >Reporter: Pablo Junge >Assignee: Nita Dembla >Priority: Major > Fix For: 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 3.2.0, > 3.1.2, 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0, > 2.0.1, 2.0.0 > > > Hive documentation for hive.stats.ndv.error does not specify that > hive.stats.ndv.error will only affect FM Sketch and not HLL. > > https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754178#comment-16754178
]
Hive QA commented on HIVE-20255:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 2
fixed = 1 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15811/dev-support/hive-personality.sh
|
| git revision | master / fd92d88 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15811/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.10.patch,
> HIVE-20255.11.patch, HIVE-20255.12.patch, HIVE-20255.13.patch,
> HIVE-20255.2.patch, HIVE-20255.3.patch, HIVE-20255.4.patch,
> HIVE-20255.5.patch, HIVE-20255.6.patch, HIVE-20255.7.patch,
> HIVE-20255.8.patch, HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Status: Open (was: Patch Available)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.10.patch,
> HIVE-20255.11.patch, HIVE-20255.12.patch, HIVE-20255.13.patch,
> HIVE-20255.2.patch, HIVE-20255.3.patch, HIVE-20255.4.patch,
> HIVE-20255.5.patch, HIVE-20255.6.patch, HIVE-20255.7.patch,
> HIVE-20255.8.patch, HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Attachment: HIVE-20255.13.patch
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.10.patch,
> HIVE-20255.11.patch, HIVE-20255.12.patch, HIVE-20255.13.patch,
> HIVE-20255.2.patch, HIVE-20255.3.patch, HIVE-20255.4.patch,
> HIVE-20255.5.patch, HIVE-20255.6.patch, HIVE-20255.7.patch,
> HIVE-20255.8.patch, HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[
https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20255:
---
Status: Patch Available (was: Open)
> Review LevelOrderWalker.java
>
>
> Key: HIVE-20255
> URL: https://issues.apache.org/jira/browse/HIVE-20255
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20255.1.patch, HIVE-20255.10.patch,
> HIVE-20255.11.patch, HIVE-20255.12.patch, HIVE-20255.13.patch,
> HIVE-20255.2.patch, HIVE-20255.3.patch, HIVE-20255.4.patch,
> HIVE-20255.5.patch, HIVE-20255.6.patch, HIVE-20255.7.patch,
> HIVE-20255.8.patch, HIVE-20255.9.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java
> * Make code more concise
> * Fix some check style issues
> {code}
> if (toWalk.get(index).getChildren() != null) {
> for(Node child : toWalk.get(index).getChildren()) {
> {code}
> Actually, the underlying implementation of {{getChildren()}} has to do some
> real work, so do not throw away the work after checking for null. Simply
> call once and store the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21171) Skip creating scratch dirs for tez if RPC is on
[ https://issues.apache.org/jira/browse/HIVE-21171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754074#comment-16754074 ] Ashutosh Chauhan commented on HIVE-21171: - +1 > Skip creating scratch dirs for tez if RPC is on > --- > > Key: HIVE-21171 > URL: https://issues.apache.org/jira/browse/HIVE-21171 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21171.1.patch, HIVE-21171.2.patch > > > There are few places e.g. during creating DAG/Vertices where scratch > directories are created for each vertex even if plan is being sent using RPC. > This adds un-necessary overhead for cloud file system e.g. S3A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754069#comment-16754069
]
Hive QA commented on HIVE-21045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956467/HIVE-21045.branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 132 failed/errored test(s), 14481 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=274)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=274)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=239)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestGetPartitionsUsingProjectionAndFilterSpecs - did not produce a TEST-*.xml
file (likely timed out) (batchId=230)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=235)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=233)
TestHiveMetaToolCommandLine - did not produce a TEST-*.xml file (likely timed
out) (batchId=235)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=235)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestMarkPartition - did not produce a
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754030#comment-16754030
]
Hive QA commented on HIVE-21045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-15810/patches/PreCommit-HIVE-Build-15810.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15810/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch, HIVE-21045.4.patch, HIVE-21045.5.patch,
> HIVE-21045.6.patch, HIVE-21045.7.patch, HIVE-21045.branch-3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754018#comment-16754018
]
Yongzhi Chen commented on HIVE-21045:
-
[~karthik.manamcheri], the tests should be kicked off automatically. Your match
name seems follows the same pattern, although a little different:
I used to name the file as HIVE-21045.01-branch-3.patch .
And it is also possible that the jira is in solved mode.
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch, HIVE-21045.4.patch, HIVE-21045.5.patch,
> HIVE-21045.6.patch, HIVE-21045.7.patch, HIVE-21045.branch-3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-21045:
Fix Version/s: (was: 4.0.0)
3.2.0
Status: Patch Available (was: Reopened)
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch, HIVE-21045.4.patch, HIVE-21045.5.patch,
> HIVE-21045.6.patch, HIVE-21045.7.patch, HIVE-21045.branch-3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen reopened HIVE-21045:
-
Open the jira to run branch-3 tests
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch, HIVE-21045.4.patch, HIVE-21045.5.patch,
> HIVE-21045.6.patch, HIVE-21045.7.patch, HIVE-21045.branch-3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[
https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753950#comment-16753950
]
Hive QA commented on HIVE-21079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956567/HIVE-21079.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15719 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNullNames[Remote]
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15809/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15809/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15809/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12956567 - PreCommit-HIVE-Build
> Replicate column statistics for partitions of partitioned Hive table.
> -
>
> Key: HIVE-21079
> URL: https://issues.apache.org/jira/browse/HIVE-21079
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch,
> HIVE-21079.03.patch, HIVE-21079.04.patch
>
>
> This task is for replicating statistics for partitions of a partitioned Hive
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[
https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753941#comment-16753941
]
Hive QA commented on HIVE-21079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 5 new + 363 unchanged - 5
fixed = 368 total (was 368) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 565
unchanged - 3 fixed = 566 total (was 568) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15809/dev-support/hive-personality.sh
|
| git revision | master / fd92d88 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15809/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15809/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15809/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql hcatalog/server-extensions
itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15809/yetus.txt |
| Powered by |
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Status: In Progress (was: Patch Available) > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Attachment: HIVE-21079.04.patch Status: Patch Available (was: In Progress) > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[
https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753896#comment-16753896
]
Hive QA commented on HIVE-21079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956563/HIVE-21079.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15808/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15808/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15808/
Messages:
{noformat}
This message was trimmed, see log for full details
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PutFileMetadataRequest.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RenamePartitionRequest.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ReplLastIdInfo.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ReplTblWriteIdStateRequest.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SchemaVersion.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TableValidWriteIds.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetAllResourcePlanResponse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetTriggersForResourePlanResponse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMValidateResourcePlanResponse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WriteNotificationLogRequest.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/UpdatePartitionColumnStatEvent.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageBuilder.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/UpdatePartitionColumnStatMessage.java:
does not exist in index
error:
[jira] [Commented] (HIVE-2722) GenericUDFUtils.findText use CharBuffer other than ByteBuffer will be better
[
https://issues.apache.org/jira/browse/HIVE-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753892#comment-16753892
]
Mani M commented on HIVE-2722:
--
HI [~pvary]
I have checked this query in HIVE 3.1.1 version and the output is
*{color:#FF}showing as 7{color}* (as given below).
So I think patch is not required.
{code:java}
hive> select instr("中文字符测试-第一行","-") from lang_det limit 1;
OK
7
Time taken: 2.683 seconds, Fetched: 1 row(s)
{code}
> GenericUDFUtils.findText use CharBuffer other than ByteBuffer will be better
>
>
> Key: HIVE-2722
> URL: https://issues.apache.org/jira/browse/HIVE-2722
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Affects Versions: 0.8.0, 0.9.0
> Environment: Linux zongren-VirtualBox 3.0.0-14-generic #23-Ubuntu SMP
> Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.7.0-cdh3u0
>Reporter: caofangkun
>Assignee: Mani M
>Priority: Minor
> Labels: udf
> Attachments: HIVE-2722.patch, udf_instr_1.q
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> select instr("中文字符测试-第一行","-") from testTable limit 1;
> result:19 (one Chinese Character was considered as 3 Unicode bits)
> select substr("中文字符测试-第一行",1,2) from testTable limit 1;
> result: "中文" (one Chinese Character was considered as 1 Unicode Unit )
> instr should considered one chinese character as one Unicode Unit too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Fix Version/s: 4.0.0 Affects Version/s: 4.0.0 Attachment: HIVE-21079.03.patch Status: Patch Available (was: In Progress) [~sankarh], PFA patch with your comments addressed. Please take a look. I have updated PR with the same. > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Status: In Progress (was: Patch Available) > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21142) Druidhandler may miss results when time constrainted by and/ors
[
https://issues.apache.org/jira/browse/HIVE-21142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753792#comment-16753792
]
Zoltan Haindrich commented on HIVE-21142:
-
tested with a local calcite snapshot that the issue is fixed by CALCITE-2802
> Druidhandler may miss results when time constrainted by and/ors
> ---
>
> Key: HIVE-21142
> URL: https://issues.apache.org/jira/browse/HIVE-21142
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: add_test.patch
>
>
> For the following query:
> {code}
> FROM druid_table_alltypesorc
> WHERE ('1968-01-01 00:00:00' <= `__time` AND `__time` <= '1970-01-01
> 00:00:00')
> OR ('1968-02-01 00:00:00' <= `__time` AND `__time` <= '1970-04-01
> 00:00:00') ORDER BY `__time` ASC LIMIT 10;
> {code}
> the druid query is:
> {code}
> druid.query.json
> {"queryType":"scan","dataSource":"default.druid_table_alltypesorc","intervals":["1900-01-01T00:00:00.000Z/1968-02-01T08:00:00.001Z"],"virtualColumns":[{"type":"expression","name":"vc","expression":"\"__time\"","outputType":"LONG"}],"columns":["vc"],"resultFormat":"compactedList"}
> {code}
> which has an invalid interval:
> {{"intervals":["1900-01-01T00:00:00.000Z/1968-02-01T08:00:00.001Z"}} which
> prevents valid results from 1969 to appear.
> note: using between the interval is handled correctly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
