[jira] [Commented] (HIVE-20394) Optimized & cleaned up HBaseQTest runner
[
https://issues.apache.org/jira/browse/HIVE-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589716#comment-16589716
]
Hive QA commented on HIVE-20394:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
44s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/util: The patch generated 5 new + 129 unchanged
- 5 fixed = 134 total (was 134) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13410/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13410/yetus/patch-mvninstall-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13410/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: hbase-handler itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13410/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimized & cleaned up HBaseQTest runner
>
>
> Key: HIVE-20394
> URL: https://issues.apache.org/jira/browse/HIVE-20394
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20394.1.patch, HIVE-20394.2.patch,
> HIVE-20394.3.patch, HIVE-20394.4.patch, HIVE-20394.5.patch
>
>
> # Set proper cluster destroy order
> # Propagated proper HBaseTestContext
> # Ported downstream fixes (CDH-63695)
> # General clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20433) Implicit String to Timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589696#comment-16589696
]
Hive QA commented on HIVE-20433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936689/HIVE-20433.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14890 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_date_format]
(batchId=61)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16]
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94]
(batchId=266)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query95]
(batchId=266)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query95]
(batchId=264)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFAddMonths.testWrongTsStr
(batchId=281)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFDateFormat.testDateFormatStr
(batchId=278)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testKillQuery (batchId=249)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13409/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13409/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13409/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936689 - PreCommit-HIVE-Build
> Implicit String to Timestamp conversion is slow
> ---
>
> Key: HIVE-20433
> URL: https://issues.apache.org/jira/browse/HIVE-20433
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch,
> HIVE-20433.3.patch, HIVE-20433.4.patch, HIVE-20433.5.patch, HIVE-20433.6.patch
>
>
> getTimestampFromString() is slow at casting dates. It throws twice before
> date conversion can happen.
>
> cc [~gopalv] [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20433) Implicit String to Timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589677#comment-16589677
]
Hive QA commented on HIVE-20433:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 2 new + 190 unchanged - 0
fixed = 192 total (was 190) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13409/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13409/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13409/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implicit String to Timestamp conversion is slow
> ---
>
> Key: HIVE-20433
> URL: https://issues.apache.org/jira/browse/HIVE-20433
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch,
> HIVE-20433.3.patch, HIVE-20433.4.patch, HIVE-20433.5.patch, HIVE-20433.6.patch
>
>
> getTimestampFromString() is slow at casting dates. It throws twice before
> date conversion can happen.
>
> cc [~gopalv] [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589668#comment-16589668
]
Hive QA commented on HIVE-18453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936672/HIVE-18453.07.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14894 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13408/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13408/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13408/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936672 - PreCommit-HIVE-Build
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch,
> HIVE-18453.06.patch, HIVE-18453.07.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20415) Hive1: Tez Session failed to return if background thread is interrupted
[
https://issues.apache.org/jira/browse/HIVE-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20415:
--
Summary: Hive1: Tez Session failed to return if background thread is
interrupted (was: Hive1: Tez Session failed to return if background thread is
intruppted)
> Hive1: Tez Session failed to return if background thread is interrupted
> ---
>
> Key: HIVE-20415
> URL: https://issues.apache.org/jira/browse/HIVE-20415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-20415-01-branch-1.patch, HIVE-20415.patch
>
>
> user canceled the query which interrupts the background thread, because of
> this interrupt background thread fail to put the session back to the pool.
> {code}
> 2018-08-14 15:55:27,581 ERROR exec.Task (TezTask.java:execute(226)) - Failed
> to execute tez graph.
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.returnSession(TezSessionPoolManager.java:176)
> {code}
> we need a similar fix as HIVE-15731
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589648#comment-16589648
]
Hive QA commented on HIVE-18453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
20s{color} | {color:red} ql generated 5 new + 2304 unchanged - 4 fixed = 2309
total (was 2308) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to LA29_5 in
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At
HiveLexer.java:org.apache.hadoop.hive.ql.parse.HiveLexer$DFA29.specialStateTransition(int,
IntStream) At HiveLexer.java:[line 12759] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA231 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46679-46692] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA274 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47029-47042] |
| | Dead store to LA28_141 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At HiveParser.java:[line 46508] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA289 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
47165-47178] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13408/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13408/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13408/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments:
[jira] [Updated] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20150: -- Attachment: HIVE-20150.10.patch > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20150.1.patch, HIVE-20150.10.patch, > HIVE-20150.2.patch, HIVE-20150.4.patch, HIVE-20150.5.patch, > HIVE-20150.6.patch, HIVE-20150.7.patch, HIVE-20150.8.patch, HIVE-20150.9.patch > > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20444) Parameter is not properly quoted in DbNotificationListener.addWriteNotificationLog
[
https://issues.apache.org/jira/browse/HIVE-20444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20444:
--
Status: Patch Available (was: Open)
> Parameter is not properly quoted in
> DbNotificationListener.addWriteNotificationLog
> --
>
> Key: HIVE-20444
> URL: https://issues.apache.org/jira/browse/HIVE-20444
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20444.1.patch
>
>
> See exception:
> {code}
> 2018-08-22T04:44:22,758 INFO [pool-8-thread-190]:
> listener.DbNotificationListener
> (DbNotificationListener.java:addWriteNotificationLog(765)) - Going to execute
> insert "WNL_WRITEID", "WNL_DATABASE", "WNL_TABLE", "WNL_PARTITION", "WNL_TABLE_OBJ",
> "WNL_PARTITION_OBJ", "WNL_FILES", "WNL_EVENT_TIME") values
> (50,124,1,'default','t1_default','','{"1":{"str":"t1_default"},"2":{"str":"default"},"3":{"str":"hrt_qa"},"4":{"i32":1534913061},"5":{"i32":0},"6":{"i32":0},"7":{"rec":{"1":{"lst":["rec",15,{"1":{"str":"t"},"2":{"str":"tinyint"}},{"1":{"str":"si"},"2":{"str":"smallint"}},{"1":{"str":"i"},"2":{"str":"int"}},{"1":{"str":"b"},"2":{"str":"bigint"}},{"1":{"str":"f"},"2":{"str":"double"}},{"1":{"str":"d"},"2":{"str":"double"}},{"1":{"str":"s"},"2":{"str":"varchar(25)"}},{"1":{"str":"dc"},"2":{"str":"decimal(38,18)"}},{"1":{"str":"bo"},"2":{"str":"varchar(5)"}},{"1":{"str":"v"},"2":{"str":"varchar(25)"}},{"1":{"str":"c"},"2":{"str":"char(25)"}},{"1":{"str":"ts"},"2":{"str":"timestamp"}},{"1":{"str":"dt"},"2":{"str":"date"}},{"1":{"str":"st"},"2":{"str":"string"}},{"1":{"str":"tz"},"2":{"str":"timestamp
> with local time
> zone('UTC')"}}]},"2":{"str":"hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default"},"3":{"str":"org.apache.hadoop.mapred.TextInputFormat"},"4":{"str":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"},"5":{"tf":0},"6":{"i32":-1},"7":{"rec":{"2":{"str":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"3":{"map":["str","str",1,{"serialization.format":"1"}]}}},"8":{"lst":["str",0]},"9":{"lst":["rec",0]},"10":{"map":["str","str",0,{}]},"11":{"rec":{"1":{"lst":["str",0]},"2":{"lst":["lst",0]},"3":{"map":["lst","str",0,{}]}}},"12":{"tf":0}}},"8":{"lst":["rec",0]},"9":{"map":["str","str",9,{"totalSize":"0","rawDataSize":"0","numRows":"0","transactional_properties":"insert_only","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"b\":\"true\",\"bo\":\"true\",\"c\":\"true\",\"d\":\"true\",\"dc\":\"true\",\"dt\":\"true\",\"f\":\"true\",\"i\":\"true\",\"s\":\"true\",\"si\":\"true\",\"st\":\"true\",\"t\":\"true\",\"ts\":\"true\",\"tz\":\"true\",\"v\":\"true\"}}","numFiles":"0","transient_lastDdlTime":"1534913062","bucketing_version":"2","transactional":"true"}]},"12":{"str":"MANAGED_TABLE"},"15":{"tf":0},"17":{"str":"hive"},"18":{"i32":1},"19":{"i64":1}}','null','hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default/delta_001_001_/00_0###delta_001_001_',1534913062)>
> 2018-08-22T04:44:22,773 ERROR [pool-8-thread-190]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) -
> MetaException(message:Unable to add write notification log
> org.postgresql.util.PSQLException: ERROR: syntax error at or near "UTC"
> Position: 1032
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
> at
> com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.addWriteNotificationLog(DbNotificationListener.java:766)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:657)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
> at
>
[jira] [Updated] (HIVE-20444) Parameter is not properly quoted in DbNotificationListener.addWriteNotificationLog
[
https://issues.apache.org/jira/browse/HIVE-20444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20444:
--
Attachment: HIVE-20444.1.patch
> Parameter is not properly quoted in
> DbNotificationListener.addWriteNotificationLog
> --
>
> Key: HIVE-20444
> URL: https://issues.apache.org/jira/browse/HIVE-20444
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20444.1.patch
>
>
> See exception:
> {code}
> 2018-08-22T04:44:22,758 INFO [pool-8-thread-190]:
> listener.DbNotificationListener
> (DbNotificationListener.java:addWriteNotificationLog(765)) - Going to execute
> insert "WNL_WRITEID", "WNL_DATABASE", "WNL_TABLE", "WNL_PARTITION", "WNL_TABLE_OBJ",
> "WNL_PARTITION_OBJ", "WNL_FILES", "WNL_EVENT_TIME") values
> (50,124,1,'default','t1_default','','{"1":{"str":"t1_default"},"2":{"str":"default"},"3":{"str":"hrt_qa"},"4":{"i32":1534913061},"5":{"i32":0},"6":{"i32":0},"7":{"rec":{"1":{"lst":["rec",15,{"1":{"str":"t"},"2":{"str":"tinyint"}},{"1":{"str":"si"},"2":{"str":"smallint"}},{"1":{"str":"i"},"2":{"str":"int"}},{"1":{"str":"b"},"2":{"str":"bigint"}},{"1":{"str":"f"},"2":{"str":"double"}},{"1":{"str":"d"},"2":{"str":"double"}},{"1":{"str":"s"},"2":{"str":"varchar(25)"}},{"1":{"str":"dc"},"2":{"str":"decimal(38,18)"}},{"1":{"str":"bo"},"2":{"str":"varchar(5)"}},{"1":{"str":"v"},"2":{"str":"varchar(25)"}},{"1":{"str":"c"},"2":{"str":"char(25)"}},{"1":{"str":"ts"},"2":{"str":"timestamp"}},{"1":{"str":"dt"},"2":{"str":"date"}},{"1":{"str":"st"},"2":{"str":"string"}},{"1":{"str":"tz"},"2":{"str":"timestamp
> with local time
> zone('UTC')"}}]},"2":{"str":"hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default"},"3":{"str":"org.apache.hadoop.mapred.TextInputFormat"},"4":{"str":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"},"5":{"tf":0},"6":{"i32":-1},"7":{"rec":{"2":{"str":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"3":{"map":["str","str",1,{"serialization.format":"1"}]}}},"8":{"lst":["str",0]},"9":{"lst":["rec",0]},"10":{"map":["str","str",0,{}]},"11":{"rec":{"1":{"lst":["str",0]},"2":{"lst":["lst",0]},"3":{"map":["lst","str",0,{}]}}},"12":{"tf":0}}},"8":{"lst":["rec",0]},"9":{"map":["str","str",9,{"totalSize":"0","rawDataSize":"0","numRows":"0","transactional_properties":"insert_only","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"b\":\"true\",\"bo\":\"true\",\"c\":\"true\",\"d\":\"true\",\"dc\":\"true\",\"dt\":\"true\",\"f\":\"true\",\"i\":\"true\",\"s\":\"true\",\"si\":\"true\",\"st\":\"true\",\"t\":\"true\",\"ts\":\"true\",\"tz\":\"true\",\"v\":\"true\"}}","numFiles":"0","transient_lastDdlTime":"1534913062","bucketing_version":"2","transactional":"true"}]},"12":{"str":"MANAGED_TABLE"},"15":{"tf":0},"17":{"str":"hive"},"18":{"i32":1},"19":{"i64":1}}','null','hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default/delta_001_001_/00_0###delta_001_001_',1534913062)>
> 2018-08-22T04:44:22,773 ERROR [pool-8-thread-190]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) -
> MetaException(message:Unable to add write notification log
> org.postgresql.util.PSQLException: ERROR: syntax error at or near "UTC"
> Position: 1032
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
> at
> com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.addWriteNotificationLog(DbNotificationListener.java:766)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:657)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore.java:7563)
>
[jira] [Assigned] (HIVE-20444) Parameter is not properly quoted in DbNotificationListener.addWriteNotificationLog
[
https://issues.apache.org/jira/browse/HIVE-20444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-20444:
-
> Parameter is not properly quoted in
> DbNotificationListener.addWriteNotificationLog
> --
>
> Key: HIVE-20444
> URL: https://issues.apache.org/jira/browse/HIVE-20444
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> See exception:
> {code}
> 2018-08-22T04:44:22,758 INFO [pool-8-thread-190]:
> listener.DbNotificationListener
> (DbNotificationListener.java:addWriteNotificationLog(765)) - Going to execute
> insert "WNL_WRITEID", "WNL_DATABASE", "WNL_TABLE", "WNL_PARTITION", "WNL_TABLE_OBJ",
> "WNL_PARTITION_OBJ", "WNL_FILES", "WNL_EVENT_TIME") values
> (50,124,1,'default','t1_default','','{"1":{"str":"t1_default"},"2":{"str":"default"},"3":{"str":"hrt_qa"},"4":{"i32":1534913061},"5":{"i32":0},"6":{"i32":0},"7":{"rec":{"1":{"lst":["rec",15,{"1":{"str":"t"},"2":{"str":"tinyint"}},{"1":{"str":"si"},"2":{"str":"smallint"}},{"1":{"str":"i"},"2":{"str":"int"}},{"1":{"str":"b"},"2":{"str":"bigint"}},{"1":{"str":"f"},"2":{"str":"double"}},{"1":{"str":"d"},"2":{"str":"double"}},{"1":{"str":"s"},"2":{"str":"varchar(25)"}},{"1":{"str":"dc"},"2":{"str":"decimal(38,18)"}},{"1":{"str":"bo"},"2":{"str":"varchar(5)"}},{"1":{"str":"v"},"2":{"str":"varchar(25)"}},{"1":{"str":"c"},"2":{"str":"char(25)"}},{"1":{"str":"ts"},"2":{"str":"timestamp"}},{"1":{"str":"dt"},"2":{"str":"date"}},{"1":{"str":"st"},"2":{"str":"string"}},{"1":{"str":"tz"},"2":{"str":"timestamp
> with local time
> zone('UTC')"}}]},"2":{"str":"hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default"},"3":{"str":"org.apache.hadoop.mapred.TextInputFormat"},"4":{"str":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"},"5":{"tf":0},"6":{"i32":-1},"7":{"rec":{"2":{"str":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"3":{"map":["str","str",1,{"serialization.format":"1"}]}}},"8":{"lst":["str",0]},"9":{"lst":["rec",0]},"10":{"map":["str","str",0,{}]},"11":{"rec":{"1":{"lst":["str",0]},"2":{"lst":["lst",0]},"3":{"map":["lst","str",0,{}]}}},"12":{"tf":0}}},"8":{"lst":["rec",0]},"9":{"map":["str","str",9,{"totalSize":"0","rawDataSize":"0","numRows":"0","transactional_properties":"insert_only","COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"b\":\"true\",\"bo\":\"true\",\"c\":\"true\",\"d\":\"true\",\"dc\":\"true\",\"dt\":\"true\",\"f\":\"true\",\"i\":\"true\",\"s\":\"true\",\"si\":\"true\",\"st\":\"true\",\"t\":\"true\",\"ts\":\"true\",\"tz\":\"true\",\"v\":\"true\"}}","numFiles":"0","transient_lastDdlTime":"1534913062","bucketing_version":"2","transactional":"true"}]},"12":{"str":"MANAGED_TABLE"},"15":{"tf":0},"17":{"str":"hive"},"18":{"i32":1},"19":{"i64":1}}','null','hdfs://mycluster/warehouse/tablespace/managed/hive/t1_default/delta_001_001_/00_0###delta_001_001_',1534913062)>
> 2018-08-22T04:44:22,773 ERROR [pool-8-thread-190]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) -
> MetaException(message:Unable to add write notification log
> org.postgresql.util.PSQLException: ERROR: syntax error at or near "UTC"
> Position: 1032
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:313)
> at
> com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:92)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.addWriteNotificationLog(DbNotificationListener.java:766)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onAcidWrite(DbNotificationListener.java:657)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$12(MetaStoreListenerNotifier.java:249)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEventWithDirectSql(MetaStoreListenerNotifier.java:305)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.addWriteNotificationLog(TxnHandler.java:1617)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addTxnWriteNotificationLog(HiveMetaStore.java:7563)
> at
>
[jira] [Commented] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589626#comment-16589626
]
Hive QA commented on HIVE-20441:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936671/HIVE-20441.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14890 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13407/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13407/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13407/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936671 - PreCommit-HIVE-Build
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 2.3.3
>
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
>
[jira] [Updated] (HIVE-20423) Set NULLS LAST as the default null ordering
[ https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20423: -- Attachment: HIVE-20423.4.patch > Set NULLS LAST as the default null ordering > --- > > Key: HIVE-20423 > URL: https://issues.apache.org/jira/browse/HIVE-20423 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch, > HIVE-20423.3.patch, HIVE-20423.4.patch > > > HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST > becomes the default null ordering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Status: Open (was: Patch Available)
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch, HIVE-19674.2.patch,
> HIVE-19674.3.patch, HIVE-19674.4.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Status: Patch Available (was: Open)
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch, HIVE-19674.2.patch,
> HIVE-19674.3.patch, HIVE-19674.4.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19674) Group by Decimal Constants push down to Druid tables.
[
https://issues.apache.org/jira/browse/HIVE-19674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19674:
---
Attachment: HIVE-19674.4.patch
> Group by Decimal Constants push down to Druid tables.
> -
>
> Key: HIVE-19674
> URL: https://issues.apache.org/jira/browse/HIVE-19674
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19674.1.patch, HIVE-19674.2.patch,
> HIVE-19674.3.patch, HIVE-19674.4.patch
>
>
> Queries like following gets generated by Tableau.
> {code}
> SELECT SUM(`ssb_druid_100`.`lo_revenue`) AS `sum_lo_revenue_ok`
> FROM `druid_ssb`.`ssb_druid_100` `ssb_druid_100`
> GROUP BY 1.1001;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589599#comment-16589599
]
Hive QA commented on HIVE-20441:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13407/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13407/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 2.3.3
>
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
>
[jira] [Updated] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Huang updated HIVE-20441:
-
Fix Version/s: 2.3.3
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 2.3.3
>
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20374) Write Hive version information to Parquet footer
[
https://issues.apache.org/jira/browse/HIVE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589563#comment-16589563
]
Hive QA commented on HIVE-20374:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/1293/HIVE-20374.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14890 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=116)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13406/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13406/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13406/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 1293 - PreCommit-HIVE-Build
> Write Hive version information to Parquet footer
>
>
> Key: HIVE-20374
> URL: https://issues.apache.org/jira/browse/HIVE-20374
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Ivanfi
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20374.01.patch, HIVE-20374.02.patch,
> HIVE-20374.03.patch, HIVE-20374.patch
>
>
> PARQUET-352 added support for the "writer.model.name" property in the Parquet
> metadata to identify the object model (application) that wrote the file.
> The easiest way to write this property is by overriding getName() of
> org.apache.parquet.hadoop.api.WriteSupport. In Hive, this would mean adding
> getName() to the
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20423) Set NULLS LAST as the default null ordering
[ https://issues.apache.org/jira/browse/HIVE-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20423: -- Attachment: HIVE-20423.3.patch > Set NULLS LAST as the default null ordering > --- > > Key: HIVE-20423 > URL: https://issues.apache.org/jira/browse/HIVE-20423 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20423.1.patch, HIVE-20423.2.patch, > HIVE-20423.3.patch > > > HIVE-20150 TopNKeyOperator pushdown can be more efficient if NULLS LAST > becomes the default null ordering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589546#comment-16589546
]
slim bouguerra commented on HIVE-20377:
---
[^HIVE-20377.15.patch] Has an updated version with Start and End Offsets
projected from input splits. The cool thing now we can see the filter
projection pushdown to trim the Kafka splits.
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.4.patch,
> HIVE-20377.5.patch, HIVE-20377.6.patch, HIVE-20377.8.patch,
> HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" :
[jira] [Updated] (HIVE-20377) Hive Kafka Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-20377:
--
Attachment: HIVE-20377.15.patch
> Hive Kafka Storage Handler
> --
>
> Key: HIVE-20377
> URL: https://issues.apache.org/jira/browse/HIVE-20377
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-20377.10.patch, HIVE-20377.11.patch,
> HIVE-20377.12.patch, HIVE-20377.15.patch, HIVE-20377.4.patch,
> HIVE-20377.5.patch, HIVE-20377.6.patch, HIVE-20377.8.patch,
> HIVE-20377.8.patch, HIVE-20377.patch
>
>
> h1. Goal
> * Read streaming data form Kafka queue as an external table.
> * Allow streaming navigation by pushing down filters on Kafka record
> partition id, offset and timestamp.
> * Insert streaming data form Kafka to an actual Hive internal table, using
> CTAS statement.
> h1. Example
> h2. Create the external table
> {code}
> CREATE EXTERNAL TABLE kafka_table (`timestamp` timestamp, page string, `user`
> string, language string, added int, deleted int, flags string,comment string,
> namespace string)
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wikipedia",
> "kafka.bootstrap.servers"="brokeraddress:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.JsonSerDe");
> {code}
> h2. Kafka Metadata
> In order to keep track of Kafka records the storage handler will add
> automatically the Kafka row metadata eg partition id, record offset and
> record timestamp.
> {code}
> DESCRIBE EXTENDED kafka_table
> timestamp timestamp from deserializer
> page string from deserializer
> user string from deserializer
> language string from deserializer
> country string from deserializer
> continent string from deserializer
> namespace string from deserializer
> newpage boolean from deserializer
> unpatrolled boolean from deserializer
> anonymous boolean from deserializer
> robot boolean from deserializer
> added int from deserializer
> deleted int from deserializer
> delta bigint from deserializer
> __partition int from deserializer
> __offset bigint from deserializer
> __timestamp bigint from deserializer
> {code}
> h2. Filter push down.
> Newer Kafka consumers 0.11.0 and higher allow seeking on the stream based on
> a given offset. The proposed storage handler will be able to leverage such
> API by pushing down filters over metadata columns, namely __partition (int),
> __offset(long) and __timestamp(long)
> For instance Query like
> {code}
> select `__offset` from kafka_table where (`__offset` < 10 and `__offset`>3
> and `__partition` = 0) or (`__partition` = 0 and `__offset` < 105 and
> `__offset` > 99) or (`__offset` = 109);
> {code}
> Will result on a scan of partition 0 only then read only records between
> offset 4 and 109.
> h2. With timestamp seeks
> The seeking based on the internal timestamps allows the handler to run on
> recently arrived data, by doing
> {code}
> select count(*) from kafka_table where `__timestamp` > 1000 *
> to_unix_timestamp(CURRENT_TIMESTAMP - interval '20' hours) ;
> {code}
> This allows for implicit relationships between event timestamps and kafka
> timestamps to be expressed in queries (i.e event_timestamp is always < than
> kafka __timestamp and kafka __timestamp is never > 15 minutes from event etc).
> h2. More examples with Avro
> {code}
> CREATE EXTERNAL TABLE wiki_kafka_avro_table
> STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
> TBLPROPERTIES
> ("kafka.topic" = "wiki_kafka_avro_table",
> "kafka.bootstrap.servers"="localhost:9092",
> "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe",
> 'avro.schema.literal'='{
> "type" : "record",
> "name" : "Wikipedia",
> "namespace" : "org.apache.hive.kafka",
> "version": "1",
> "fields" : [ {
> "name" : "isrobot",
> "type" : "boolean"
> }, {
> "name" : "channel",
> "type" : "string"
> }, {
> "name" : "timestamp",
> "type" : "string"
> }, {
> "name" : "flags",
> "type" : "string"
> }, {
> "name" : "isunpatrolled",
> "type" : "boolean"
> }, {
> "name" : "page",
> "type" :
[jira] [Commented] (HIVE-20374) Write Hive version information to Parquet footer
[
https://issues.apache.org/jira/browse/HIVE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589538#comment-16589538
]
Hive QA commented on HIVE-20374:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13406/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13406/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Write Hive version information to Parquet footer
>
>
> Key: HIVE-20374
> URL: https://issues.apache.org/jira/browse/HIVE-20374
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Ivanfi
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20374.01.patch, HIVE-20374.02.patch,
> HIVE-20374.03.patch, HIVE-20374.patch
>
>
> PARQUET-352 added support for the "writer.model.name" property in the Parquet
> metadata to identify the object model (application) that wrote the file.
> The easiest way to write this property is by overriding getName() of
> org.apache.parquet.hadoop.api.WriteSupport. In Hive, this would mean adding
> getName() to the
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18565) Set some timeout to all CliDriver tests
[
https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589496#comment-16589496
]
Hive QA commented on HIVE-18565:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936663/HIVE-18565.06.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 14699 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=264)
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=151)
[mapreduce2.q,orc_llap_counters1.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,parquet_struct_type_vectorization.q,orc_merge1.q,parquet_types_vectorization.q,orc_merge_diff_fs.q,llap_stats.q,llapdecider.q,llap_nullscan.q,orc_ppd_basic.q,rcfile_merge4.q,orc_merge3.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=153)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,table_nonprintable.q,tez_union_dynamic_partition_2.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[import_exported_table.q,tez_union_dynamic_partition.q,except_distinct.q,orc_merge10.q,mapreduce1.q,mm_all.q,orc_merge4.q,dynamic_semijoin_user_level.q,temp_table_external.q,orc_struct_type_vectorization.q,llap_udf.q,schemeAuthority.q,external_table_with_space_in_location_path.q,external_table_purge.q,remote_script.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=155)
[parquet_complex_types_vectorization.q,cte_4.q,file_with_header_footer.q,explainuser_2.q,rcfile_merge2.q,bucket5.q,parquet_map_type_vectorization.q,insert_into2.q,reduce_deduplicate.q,orc_llap_counters.q,schemeAuthority2.q,load_fs2.q,reduce_deduplicate_distinct.q,rcfile_merge3.q,intersect_distinct.q]
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=107)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,topnkey.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,hybridgrace_hashjoin_1.q,delete_orig_table.q,hybridgrace_hashjoin_2.q,tez-tag.q,vector_topnkey.q,explainanalyze_5.q,vector_join_part_col_char.q]
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_with_masking]
(batchId=175)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13405/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13405/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13405/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936663 - PreCommit-HIVE-Build
> Set some timeout to all CliDriver tests
> ---
>
> Key: HIVE-18565
> URL: https://issues.apache.org/jira/browse/HIVE-18565
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18565.01.patch, HIVE-18565.02.patch,
> HIVE-18565.03.patch, HIVE-18565.04.patch, HIVE-18565.05.patch,
> HIVE-18565.05.patch, HIVE-18565.06.patch
>
>
> if a testcase runs into an infinite loop or something ; it should still fail
> instead of causing timeout at the ptest executor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16608) Incorrect results for INNER JOIN ON clause / WHERE involving NVL / COALESCE
[
https://issues.apache.org/jira/browse/HIVE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589476#comment-16589476
]
Sathyaprakash Govindasamy commented on HIVE-16608:
--
Another work around is to write query like this
where nvl(TABLEA.attr, 0)= CASE WHEN TABLEB.attr IS NOT NULL THEN TABLEB.attr
ELSE 0 END
Converting one of the NVL statement to CASE statement is enough
> Incorrect results for INNER JOIN ON clause / WHERE involving NVL / COALESCE
> ---
>
> Key: HIVE-16608
> URL: https://issues.apache.org/jira/browse/HIVE-16608
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 2.1.1
>Reporter: Dudu Markovitz
>Priority: Major
>
> {code}
> create table TABLEA (key int,attr int);
> create table TABLEB (key int,attr int);
> insert into TABLEA values (1,null),(2,0),(3,null);
> insert into TABLEB values (1,null),(2,null),(3,0);
> {code}
> The following SELECT statements should return 3 rows but return none:
> {code}
> select TABLEA.attr
> from TABLEA join TABLEB
> on TABLEA.key=TABLEB.key
> and nvl(TABLEA.attr, 0)=nvl(TABLEB.attr, 0)
> {code}
> {code}
> select TABLEA.attr
> from TABLEA join TABLEB
> on TABLEA.key=TABLEB.key
> where nvl(TABLEA.attr, 0)=nvl(TABLEB.attr, 0)
> {code}
> The same results returned for COALESCE
> In the execution plan we can notice the wrong predicate *attr is not null*,
> twice
> {code}
> predicate: (key is not null and attr is not null) (type: boolean)
> {code}
> Selecting columns from both tables seems to prevent the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589464#comment-16589464
]
Vineet Garg commented on HIVE-18873:
[~an...@apache.org] Can you upload rebased patch? With new policy patch now
need green run for tests.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18565) Set some timeout to all CliDriver tests
[
https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589457#comment-16589457
]
Hive QA commented on HIVE-18565:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13405/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13405/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set some timeout to all CliDriver tests
> ---
>
> Key: HIVE-18565
> URL: https://issues.apache.org/jira/browse/HIVE-18565
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18565.01.patch, HIVE-18565.02.patch,
> HIVE-18565.03.patch, HIVE-18565.04.patch, HIVE-18565.05.patch,
> HIVE-18565.05.patch, HIVE-18565.06.patch
>
>
> if a testcase runs into an infinite loop or something ; it should still fail
> instead of causing timeout at the ptest executor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20396) Test HS2 open_connection metrics
[
https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589422#comment-16589422
]
Hive QA commented on HIVE-20396:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936658/HIVE-20396.06.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14892 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsHttp.testOpenConnectionMetrics
(batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13404/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13404/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13404/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936658 - PreCommit-HIVE-Build
> Test HS2 open_connection metrics
>
>
> Key: HIVE-20396
> URL: https://issues.apache.org/jira/browse/HIVE-20396
> Project: Hive
> Issue Type: Test
> Components: HiveServer2
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20396.01.patch, HIVE-20396.02.patch,
> HIVE-20396.03.patch, HIVE-20396.04.patch, HIVE-20396.05.patch,
> HIVE-20396.06.patch, HIVE-20396.patch
>
>
> HiveServer2 is emitting metrics _default.General.open_connections_ in both
> binary and http mode. These metrics should be tested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded
[ https://issues.apache.org/jira/browse/HIVE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589408#comment-16589408 ] Sergey Shelukhin commented on HIVE-20443: - [~thejas] can you take a look? > txn stats cleanup in compaction txn handler is unneeded > --- > > Key: HIVE-20443 > URL: https://issues.apache.org/jira/browse/HIVE-20443 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20443.patch > > > This is handled via write ID being invalid for the stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded
[ https://issues.apache.org/jira/browse/HIVE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20443: Attachment: HIVE-20443.patch > txn stats cleanup in compaction txn handler is unneeded > --- > > Key: HIVE-20443 > URL: https://issues.apache.org/jira/browse/HIVE-20443 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20443.patch > > > This is handled via write ID being invalid for the stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded
[ https://issues.apache.org/jira/browse/HIVE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20443: Status: Patch Available (was: Open) > txn stats cleanup in compaction txn handler is unneeded > --- > > Key: HIVE-20443 > URL: https://issues.apache.org/jira/browse/HIVE-20443 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20443.patch > > > This is handled via write ID being invalid for the stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded
[ https://issues.apache.org/jira/browse/HIVE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20443: Description: This is handled via write ID being invalid for the stats. > txn stats cleanup in compaction txn handler is unneeded > --- > > Key: HIVE-20443 > URL: https://issues.apache.org/jira/browse/HIVE-20443 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > This is handled via write ID being invalid for the stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20443) txn stats cleanup in compaction txn handler is unneeded
[ https://issues.apache.org/jira/browse/HIVE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20443: --- > txn stats cleanup in compaction txn handler is unneeded > --- > > Key: HIVE-20443 > URL: https://issues.apache.org/jira/browse/HIVE-20443 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20442) Hive stale lock when the hiveserver2 background thread died abruptly
[
https://issues.apache.org/jira/browse/HIVE-20442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh reassigned HIVE-20442:
-
Assignee: Rajkumar Singh
> Hive stale lock when the hiveserver2 background thread died abruptly
>
>
> Key: HIVE-20442
> URL: https://issues.apache.org/jira/browse/HIVE-20442
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 2.1.1
> Environment: Hive-2.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> this look like a race condition where background thread is not able to
> release the lock it aquired.
> 1. hiveserver2 background thread request for lock
> {code}
> 2018-08-20T14:13:38,813 INFO [HiveServer2-Background-Pool: Thread-X]:
> lockmgr.DbLockManager (DbLockManager.java:lock(100)) - Requesting:
> queryId=hive_xxx LockRequest(component:[LockComponent(type:SHARED_READ,
> level:TABLE, dbname:testdb, tablename:test_table, operationType:SELECT)],
> txnid:0, user:hive, hostname:HOSTNAME, agentInfo:hive_xxx)
> {code}
> 2. acquired the lock and start heartbeating
> {code}
> 2018-08-20T14:36:30,233 INFO [HiveServer2-Background-Pool: Thread-X]:
> lockmgr.DbTxnManager (DbTxnManager.java:startHeartbeat(517)) - Started
> heartbeat with delay/interval = 15/15 MILLISECONDS for
> query: agentInfo:hive_xxx
> {code}
> 3. during time between event #1 and #2, client disconnected and deleteContext
> cleanup the session dir
> {code}
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-XXX]:
> thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(136)) -
> Session disconnected without closing properly.
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-]:
> thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(140)) -
> Closing the session: SessionHandle [3be07faf-5544-4178-8b50-8173002b171a]
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-]:
> service.CompositeService (SessionManager.java:closeSession(363)) - Session
> closed, SessionHandle [xxx], current sessions:2
> {code}
> 4. background thread died with NPE while trying to get the queryid
> {code}
> java.lang.NullPointerException: null
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1568)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_77]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_77]
> {code}
> did not get a chance to release the lock and heartbeater thread continue
> heartbeat indefinately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20442) Hive stale lock when the hiveserver2 background thread died with NPE
[
https://issues.apache.org/jira/browse/HIVE-20442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20442:
--
Summary: Hive stale lock when the hiveserver2 background thread died with
NPE (was: Hive stale lock when the hiveserver2 background thread died abruptly)
> Hive stale lock when the hiveserver2 background thread died with NPE
>
>
> Key: HIVE-20442
> URL: https://issues.apache.org/jira/browse/HIVE-20442
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Affects Versions: 2.1.1
> Environment: Hive-2.1
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> this look like a race condition where background thread is not able to
> release the lock it aquired.
> 1. hiveserver2 background thread request for lock
> {code}
> 2018-08-20T14:13:38,813 INFO [HiveServer2-Background-Pool: Thread-X]:
> lockmgr.DbLockManager (DbLockManager.java:lock(100)) - Requesting:
> queryId=hive_xxx LockRequest(component:[LockComponent(type:SHARED_READ,
> level:TABLE, dbname:testdb, tablename:test_table, operationType:SELECT)],
> txnid:0, user:hive, hostname:HOSTNAME, agentInfo:hive_xxx)
> {code}
> 2. acquired the lock and start heartbeating
> {code}
> 2018-08-20T14:36:30,233 INFO [HiveServer2-Background-Pool: Thread-X]:
> lockmgr.DbTxnManager (DbTxnManager.java:startHeartbeat(517)) - Started
> heartbeat with delay/interval = 15/15 MILLISECONDS for
> query: agentInfo:hive_xxx
> {code}
> 3. during time between event #1 and #2, client disconnected and deleteContext
> cleanup the session dir
> {code}
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-XXX]:
> thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(136)) -
> Session disconnected without closing properly.
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-]:
> thrift.ThriftCLIService (ThriftBinaryCLIService.java:deleteContext(140)) -
> Closing the session: SessionHandle [3be07faf-5544-4178-8b50-8173002b171a]
> 2018-08-21T15:39:57,820 INFO [HiveServer2-Handler-Pool: Thread-]:
> service.CompositeService (SessionManager.java:closeSession(363)) - Session
> closed, SessionHandle [xxx], current sessions:2
> {code}
> 4. background thread died with NPE while trying to get the queryid
> {code}
> java.lang.NullPointerException: null
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1568)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1211)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1204)
> ~[hive-exec-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> [hive-service-2.1.0.2.6.5.0-292.jar:2.1.0.2.6.5.0-292]
> at java.security.AccessController.doPrivileged(Native Method)
> [?:1.8.0_77]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_77]
> {code}
> did not get a chance to release the lock and heartbeater thread continue
> heartbeat indefinately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20396) Test HS2 open_connection metrics
[
https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589382#comment-16589382
]
Hive QA commented on HIVE-20396:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13404/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13404/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Test HS2 open_connection metrics
>
>
> Key: HIVE-20396
> URL: https://issues.apache.org/jira/browse/HIVE-20396
> Project: Hive
> Issue Type: Test
> Components: HiveServer2
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20396.01.patch, HIVE-20396.02.patch,
> HIVE-20396.03.patch, HIVE-20396.04.patch, HIVE-20396.05.patch,
> HIVE-20396.06.patch, HIVE-20396.patch
>
>
> HiveServer2 is emitting metrics _default.General.open_connections_ in both
> binary and http mode. These metrics should be tested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589376#comment-16589376 ] Vihang Karajgaonkar commented on HIVE-20306: [~akolb] [~pvary] Can you please review? > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589365#comment-16589365
]
Hive QA commented on HIVE-20440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936698/HIVE-20440.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14890 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13403/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13403/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13403/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936698 - PreCommit-HIVE-Build
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch, HIVE-20440.04.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589329#comment-16589329
]
Hive QA commented on HIVE-20440:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 5 unchanged - 2
fixed = 5 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} ql generated 0 new + 2307 unchanged - 1 fixed = 2307
total (was 2308) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13403/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13403/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch, HIVE-20440.04.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20439) Use the inflated memory limit during join selection for llap
[
https://issues.apache.org/jira/browse/HIVE-20439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589320#comment-16589320
]
Hive QA commented on HIVE-20439:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936643/HIVE-20439.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14890 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_6]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join32_lessSize]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_llap]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=164)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13402/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13402/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13402/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936643 - PreCommit-HIVE-Build
> Use the inflated memory limit during join selection for llap
>
>
> Key: HIVE-20439
> URL: https://issues.apache.org/jira/browse/HIVE-20439
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20439.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20430) CachedStore: bug fixes for TestEmbeddedHiveMetaStore & TestRemoteHiveMetaStore
[ https://issues.apache.org/jira/browse/HIVE-20430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-20430: Summary: CachedStore: bug fixes for TestEmbeddedHiveMetaStore & TestRemoteHiveMetaStore (was: CachedStore: getTableObjectsByName incorrectly adds a null object to the table list and does not throw the expected UnknownDBException) > CachedStore: bug fixes for TestEmbeddedHiveMetaStore & TestRemoteHiveMetaStore > -- > > Key: HIVE-20430 > URL: https://issues.apache.org/jira/browse/HIVE-20430 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.04.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20439) Use the inflated memory limit during join selection for llap
[
https://issues.apache.org/jira/browse/HIVE-20439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589295#comment-16589295
]
Hive QA commented on HIVE-20439:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 45 unchanged - 8 fixed
= 46 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13402/dev-support/hive-personality.sh
|
| git revision | master / 236a32c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13402/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13402/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use the inflated memory limit during join selection for llap
>
>
> Key: HIVE-20439
> URL: https://issues.apache.org/jira/browse/HIVE-20439
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20439.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Description: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions See this [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] Suppose A is the set of all ValidTxnList across all active readers. Each ValidTxnList has minOpenTxnId. MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all currently active readers This means that no active transaction in the system sees any txn with txnid < X as open. This means if construct ValidTxnIdList with HWM=X-1 and use that in getAcidState(), any files determined by this call as 'obsolete', will be seen as obsolete by any existing/future reader, i.e. can be physically deleted. This is also necessary for multi-statement transactions where relying on the state of Lock Manager is not sufficient. For example Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. 13 commits (via it's parent txn) at t2 > t1. (17 is still running). Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on Table1/Part1 (17 is still running) Now delta_13 may be cleaned since it can be seen as obsolete and there may be no locks on it, i.e. no one is reading it. Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot use base_14 is that may have absorbed delete events from delete_delta_14. Using MIN_HISTORY_LEVEL solves this. See description of HIVE-18747 for more details on MIN_HISTORY_LEVEL was: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions See this [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] Suppose A is the set of all ValidTxnList across all active readers. Each ValidTxnList has minOpenTxnId. MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all currently active readers This means that no active transaction in the system sees any txn with txnid < X as open. This means if construct ValidTxnIdList with HWM=X-1 and use that in getAcidState(), any files determined by this call as 'obsolete', will be seen as obsolete by any existing/future reader, i.e. can be physically deleted. This is also necessary for multi-statement transactions where relying on the state of Lock Manager is not sufficient. For example Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. 13 commits (via it's parent txn) at t2 > t1. (17 is still running). Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on Table1/Part1 (17 is still running) Now delta_13 may be cleaned since it can be seen as obsolete and there may be no locks on it, i.e. no one is reading it. Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot use base_14 is that may have absorbed delete events from delete_delta_14. Using MIN_HISTORY_LEVEL solves this. > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Description: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions See this [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] Suppose A is the set of all ValidTxnList across all active readers. Each ValidTxnList has minOpenTxnId. MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all currently active readers This means that no active transaction in the system sees any txn with txnid < X as open. This means if construct ValidTxnIdList with HWM=X-1 and use that in getAcidState(), any files determined by this call as 'obsolete', will be seen as obsolete by any existing/future reader, i.e. can be physically deleted. This is also necessary for multi-statement transactions where relying on the state of Lock Manager is not sufficient. For example Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. 13 commits (via it's parent txn) at t2 > t1. (17 is still running). Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on Table1/Part1 (17 is still running) Now delta_13 may be cleaned since it can be seen as obsolete and there may be no locks on it, i.e. no one is reading it. Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot use base_14 is that may have absorbed delete events from delete_delta_14. Using MIN_HISTORY_LEVEL solves this. was: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Using MIN_HISTORY_LEVEL solves this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Target Version/s: 4.0.0 (was: 3.0.0) > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Using MIN_HISTORY_LEVEL solves this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589267#comment-16589267
]
Hive QA commented on HIVE-20440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936654/HIVE-20440.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14889 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13401/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13401/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13401/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936654 - PreCommit-HIVE-Build
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18772) Make Acid Cleaner use MIN_HISTORY_LEVEL
[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18772: -- Description: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions was: Instead of using Lock Manager state as it currently does. This will eliminate possible race conditions > Make Acid Cleaner use MIN_HISTORY_LEVEL > --- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20418) LLAP IO may not handle ORC files that have row index disabled correctly for queries with no columns selected
[ https://issues.apache.org/jira/browse/HIVE-20418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20418: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > LLAP IO may not handle ORC files that have row index disabled correctly for > queries with no columns selected > > > Key: HIVE-20418 > URL: https://issues.apache.org/jira/browse/HIVE-20418 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20418.01.patch, HIVE-20418.02.patch, > HIVE-20418.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20418) LLAP IO may not handle ORC files that have row index disabled correctly for queries with no columns selected
[ https://issues.apache.org/jira/browse/HIVE-20418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20418: Summary: LLAP IO may not handle ORC files that have row index disabled correctly for queries with no columns selected (was: LLAP IO may not handle ORC files with row index disabled correctly for queries with no columns selected) > LLAP IO may not handle ORC files that have row index disabled correctly for > queries with no columns selected > > > Key: HIVE-20418 > URL: https://issues.apache.org/jira/browse/HIVE-20418 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20418.01.patch, HIVE-20418.02.patch, > HIVE-20418.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589233#comment-16589233
]
Hive QA commented on HIVE-20440:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 5 unchanged - 2 fixed
= 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} ql generated 0 new + 2307 unchanged - 1 fixed = 2307
total (was 2308) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13401/dev-support/hive-personality.sh
|
| git revision | master / ff25543 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13401/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13401/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20431) txn stats write ID check triggers on set location
[ https://issues.apache.org/jira/browse/HIVE-20431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20431: Attachment: HIVE-20431.02.patch > txn stats write ID check triggers on set location > - > > Key: HIVE-20431 > URL: https://issues.apache.org/jira/browse/HIVE-20431 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20431.01.patch, HIVE-20431.02.patch, > HIVE-20431.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20409) Hive ACID: Update/delete/merge does not clean hdfs staging directory
[ https://issues.apache.org/jira/browse/HIVE-20409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20409: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~Rajkumar Singh] > Hive ACID: Update/delete/merge does not clean hdfs staging directory > > > Key: HIVE-20409 > URL: https://issues.apache.org/jira/browse/HIVE-20409 > Project: Hive > Issue Type: Bug > Components: Transactions > Environment: Hive-2.1,java-1.8 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20409.patch > > > UpdateDeleteSemanticAnalyzer creates query context while rewriting the > context which doesn't set hdfscleanup, As a result, Driver doesn't clear the > staging dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20409) Hive ACID: Update/delete/merge does not clean hdfs staging directory
[ https://issues.apache.org/jira/browse/HIVE-20409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589223#comment-16589223 ] Vineet Garg commented on HIVE-20409: +1 > Hive ACID: Update/delete/merge does not clean hdfs staging directory > > > Key: HIVE-20409 > URL: https://issues.apache.org/jira/browse/HIVE-20409 > Project: Hive > Issue Type: Bug > Components: Transactions > Environment: Hive-2.1,java-1.8 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-20409.patch > > > UpdateDeleteSemanticAnalyzer creates query context while rewriting the > context which doesn't set hdfscleanup, As a result, Driver doesn't clear the > staging dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589204#comment-16589204
]
Hive QA commented on HIVE-17084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936641/HIVE-170884.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 77 failed/errored test(s), 14884 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_SortUnionTransposeRule]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer8]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join39] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join40] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join46] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join47] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join6] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_emit_interval]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lineage1] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_4]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part4]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin46] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optional_outer]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_46]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notexists]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notexists_having]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin_having]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union22] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_lateralview]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_3]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_mapjoin]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join0]
(batchId=65)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_7]
(batchId=188)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
(batchId=188)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join6]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join39]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join40]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join6] (batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=117)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589203#comment-16589203
]
Hive QA commented on HIVE-17084:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} contrib in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13400/dev-support/hive-personality.sh
|
| git revision | master / 4408661 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common accumulo-handler contrib hbase-handler
itests/hive-blobstore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13400/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.1.patch, HIVE-170884.4.patch,
> HIVE-170884.5.patch, HIVE-170884.7.patch
>
>
> This flag is off by default and could result in bad plans due to missing
> column statistics.
--
This message was sent by Atlassian
[jira] [Updated] (HIVE-20433) Implicit String to Timestamp conversion is slow
[ https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20433: -- Attachment: HIVE-20433.6.patch > Implicit String to Timestamp conversion is slow > --- > > Key: HIVE-20433 > URL: https://issues.apache.org/jira/browse/HIVE-20433 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch, > HIVE-20433.3.patch, HIVE-20433.4.patch, HIVE-20433.5.patch, HIVE-20433.6.patch > > > getTimestampFromString() is slow at casting dates. It throws twice before > date conversion can happen. > > cc [~gopalv] [~ashutoshc] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20433) Implicit String to Timestamp conversion is slow
[ https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20433: -- Attachment: HIVE-20433.5.patch > Implicit String to Timestamp conversion is slow > --- > > Key: HIVE-20433 > URL: https://issues.apache.org/jira/browse/HIVE-20433 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch, > HIVE-20433.3.patch, HIVE-20433.4.patch, HIVE-20433.5.patch > > > getTimestampFromString() is slow at casting dates. It throws twice before > date conversion can happen. > > cc [~gopalv] [~ashutoshc] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20237) Do Not Print StackTraces to STDERR in HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-20237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-20237:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~afan]
> Do Not Print StackTraces to STDERR in HiveMetaStore
> ---
>
> Key: HIVE-20237
> URL: https://issues.apache.org/jira/browse/HIVE-20237
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Trivial
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20237.1.patch, HIVE-20237.2.patch
>
>
> {code:java|title=HiveMetaStore.java}
> } catch (Throwable x) {
> x.printStackTrace();
> HMSHandler.LOG.error(StringUtils.stringifyException(x));
> throw x;
> }
> {code}
> Bad design here of "log and throw". Don't do it. Just throw the exception
> and let it be handled, and logged, in one place. At the very least, we don't
> need the error message to go into the STDERR logs with {{printStackTrace}},
> please remove. And remove the {{stringifyException}} code. Just use the
> normal logging faciltiies at the 'debug' level logging to hide the stack
> trace during normal operations.
> {code}
> HMSHandler.LOG.debug("Error", e);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20246) Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
[
https://issues.apache.org/jira/browse/HIVE-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-20246:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~afan]
> Configurable collecting stats by using DO_NOT_UPDATE_STATS table property
> -
>
> Key: HIVE-20246
> URL: https://issues.apache.org/jira/browse/HIVE-20246
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20246.5.patch, HIVE-20246.6.patch,
> HIVE-20246.7.patch
>
>
> By default, Hive collects stats when running operations like alter table
> partition(s), create table, and create external table. However, collecting
> stats requires Metastore lists all files under the table directory and the
> file listing operation can be very expensive particularly on filesystems like
> S3.
> HIVE-18743 made DO_NOT_UPDATE_STATS table property could be selectively
> prevent stats collection.
> This Jira aims at introducing DO_NOT_UPDATE_STATS table property into the
> MetaStoreUtils.updatePartitionStatsFast. By adding this, user can be
> selectively prevent stats collection when doing alter table partition(s)
> operation at table level. For example, set 'Alter Table S3_Table set
> tblproperties('DO_NOT_UPDATE_STATS'='TRUE');' MetaStore will not collect
> stats for the specified S3_Table when alter table add partition(key1=val1,
> key2=val2);
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19814) RPC Server port is always random for spark
[
https://issues.apache.org/jira/browse/HIVE-19814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589134#comment-16589134
]
Sahil Takiar commented on HIVE-19814:
-
[~bharos92] the new unit test looks good. Just one minor comment, inside
{{testServerPortAssignment}} it would be good to open a new {{SparkSession}}
using the {{SparkSessionManager}}. This should help verify that the new
{{SparkSession}} is successfully able to connect to a {{RpcServer}} with a
custom port.
> RPC Server port is always random for spark
> --
>
> Key: HIVE-19814
> URL: https://issues.apache.org/jira/browse/HIVE-19814
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.3.0, 3.0.0, 2.4.0, 4.0.0
>Reporter: bounkong khamphousone
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19814.1.patch, HIVE-19814.2.patch
>
>
> RPC server port is always a random one. In fact, the problem is in
> RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include
> SPARK_RPC_SERVER_PORT.
>
> I've found this issue while trying to make hive-on-spark running inside
> docker.
>
> HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf
> > SparkSessionManagerImpl.setup and the latter call
> SparkClientFactory.initialize(conf) which initialize the rpc server. This
> RPCServer is then used to create the sparkClient which use the rpc server
> port as --remote-port arg. Since initiateSparkConf ignore
> SPARK_RPC_SERVER_PORT, then it will always be a random port.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20394) Optimized & cleaned up HBaseQTest runner
[ https://issues.apache.org/jira/browse/HIVE-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20394: -- Attachment: HIVE-20394.5.patch > Optimized & cleaned up HBaseQTest runner > > > Key: HIVE-20394 > URL: https://issues.apache.org/jira/browse/HIVE-20394 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 4.0.0 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20394.1.patch, HIVE-20394.2.patch, > HIVE-20394.3.patch, HIVE-20394.4.patch, HIVE-20394.5.patch > > > # Set proper cluster destroy order > # Propagated proper HBaseTestContext > # Ported downstream fixes (CDH-63695) > # General clean up -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18624) Parsing time is extremely high (~10 min) for queries with complex select expressions
[
https://issues.apache.org/jira/browse/HIVE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18624:
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
2.4.0
Target Version/s: 3.1.0, 2.4.0, 4.0.0 (was: 2.4.0, 3.1.0, 4.0.0)
Status: Resolved (was: Patch Available)
pushed to master,branch-3,branch-2. Thank you Ashutosh for reviewing the
changes!
> Parsing time is extremely high (~10 min) for queries with complex select
> expressions
>
>
> Key: HIVE-18624
> URL: https://issues.apache.org/jira/browse/HIVE-18624
> Project: Hive
> Issue Type: Bug
> Components: Hive, Parser
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Amruth S
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 3.2.0
>
> Attachments: HIVE-18624.01.patch, HIVE-18624.02.patch, thread_dump
>
>
> Explain of the same query takes
> 0.1 to 3 seconds in hive 2.1.0 &
> 10-15 min in hive 2.3.2 & latest master
> Sample expression below
> {code:java}
> EXPLAIN
> SELECT DISTINCT
> IF(lower('a') <= lower('a')
> ,'a'
> ,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
> ,'a'
> ,IF(if('a' = 'a', TRUE, FALSE) = 1
> ,'a'
> ,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
>and lower(if('a' = 'a','a','a')) <= lower('a'))
> OR ('a' like 'a' OR 'a' like 'a')
> OR 'a' in ('a','a')
> ,'a'
> ,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
> ,'a'
> ,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as
> string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp()
> <= unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) +
> 9*3600
> ,'a'
> ,If(lower('a') <= lower('a')
> and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
> ,'a'
> ,IF('a'=1 AND 'a'=1
> ,'a'
> ,IF('a' = 1 and COALESCE(cast('a' as int),0) = 0
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,If('a' = 'a' AND
> lower('a')>lower(if(lower('a')<1830,'a',cast(date_add('a',1) as timestamp)))
> ,'a'
> ,IF('a' = 1
> ,IF('a' in ('a', 'a') and ((unix_timestamp('a')-unix_timestamp('a')) / 60)
> > 30 and 'a' = 1
> ,'a', 'a')
> ,IF(if('a' = 'a', FALSE, TRUE ) = 1 AND 'a' IS NULL
> ,'a'
> ,IF('a' = 1 and 'a'>0
> , 'a'
> ,IF('a' = 1 AND 'a' ='a'
> ,'a'
> ,IF('a' is not null and 'a' is not null and 'a' > 'a'
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,If('a' = 1
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' ='a' and 'a' ='a' and cast(unix_timestamp('a') as int) + 93600 <
> cast(unix_timestamp() as int)
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,IF('a' = 'a' and 'a' in ('a','a','a')
> ,'a'
> ,IF('a' = 'a'
> ,'a','a'))
> )))
> AS test_comp_exp
> {code}
>
> Taking a look at [^thread_dump] shows a very large function stack getting
> created.
> Reverting HIVE-15578 (92f31d07aa988d4a460aac56e369bfa386361776) seem to speed
> up the parsing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20394) Optimized & cleaned up HBaseQTest runner
[
https://issues.apache.org/jira/browse/HIVE-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589118#comment-16589118
]
Hive QA commented on HIVE-20394:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936624/HIVE-20394.4.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 203 failed/errored test(s), 9941 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=256)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
(batchId=267)
org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreNegativeCliDriver
(batchId=267)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=47)
[jira] [Updated] (HIVE-19069) Hive can't read int32 and int64 Parquet decimal values
[
https://issues.apache.org/jira/browse/HIVE-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated HIVE-19069:
---
Description:
Parquet supports several minor types for Decimal ligical data type:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
But Hive supports only "fixed_len_byte_array":
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java#L335
After creating parquet external table and quering it via Hive:
{code}
hive> select * from decimal_parquet;
OK
Failed with exception
java.io.IOException:org.apache.parquet.io.ParquetDecodingException: Can not
read value at 1 in block 0 in file maprfs:///tmp/decimal_parquet/0_0_0.parquet
{code}
The sample of parquet file with decimal int32 values is added to the jira:
{code}
vitalii@vitalii-pc:~$ java -jar
parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
schema /tmp/decimal_parquet/0_0_0.parquet
message root {
optional binary a (UTF8);
optional int32 b (DECIMAL(7,2));
}
vitalii@vitalii-pc:~$ java -jar
parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
cat /tmp/md4107_par/0_0_0.parquet
a = a
b = 100
{code}
was:
Parquet supports several minor types for Decimal ligical data type:
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
But Hive supports only "fixed_len_byte_array":
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java#L423
After creating parquet external table and quering it via Hive:
{code}
hive> select * from decimal_parquet;
OK
Failed with exception
java.io.IOException:org.apache.parquet.io.ParquetDecodingException: Can not
read value at 1 in block 0 in file maprfs:///tmp/decimal_parquet/0_0_0.parquet
{code}
The sample of parquet file with decimal int32 values is added to the jira:
{code}
vitalii@vitalii-pc:~$ java -jar
parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
schema /tmp/decimal_parquet/0_0_0.parquet
message root {
optional binary a (UTF8);
optional int32 b (DECIMAL(7,2));
}
vitalii@vitalii-pc:~$ java -jar
parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
cat /tmp/md4107_par/0_0_0.parquet
a = a
b = 100
{code}
> Hive can't read int32 and int64 Parquet decimal values
> --
>
> Key: HIVE-19069
> URL: https://issues.apache.org/jira/browse/HIVE-19069
> Project: Hive
> Issue Type: Improvement
> Components: Types
>Affects Versions: 2.3.2
>Reporter: Vitalii Diravka
>Priority: Major
> Labels: decimal, parquet
> Attachments: 0_0_0.parquet
>
>
> Parquet supports several minor types for Decimal ligical data type:
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
> But Hive supports only "fixed_len_byte_array":
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java#L335
> After creating parquet external table and quering it via Hive:
> {code}
> hive> select * from decimal_parquet;
> OK
> Failed with exception
> java.io.IOException:org.apache.parquet.io.ParquetDecodingException: Can not
> read value at 1 in block 0 in file maprfs:///tmp/decimal_parquet/0_0_0.parquet
> {code}
> The sample of parquet file with decimal int32 values is added to the jira:
> {code}
> vitalii@vitalii-pc:~$ java -jar
> parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
> schema /tmp/decimal_parquet/0_0_0.parquet
> message root {
> optional binary a (UTF8);
> optional int32 b (DECIMAL(7,2));
> }
> vitalii@vitalii-pc:~$ java -jar
> parquet-tools/parquet-mr/parquet-tools/target/parquet-tools-1.6.0rc3-SNAPSHOT.jar
> cat /tmp/md4107_par/0_0_0.parquet
> a = a
> b = 100
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20394) Optimized & cleaned up HBaseQTest runner
[
https://issues.apache.org/jira/browse/HIVE-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589107#comment-16589107
]
Hive QA commented on HIVE-20394:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/util: The patch generated 5 new + 129 unchanged
- 5 fixed = 134 total (was 134) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13399/dev-support/hive-personality.sh
|
| git revision | master / 09c6360 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13399/yetus/patch-mvninstall-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13399/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: hbase-handler itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13399/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimized & cleaned up HBaseQTest runner
>
>
> Key: HIVE-20394
> URL: https://issues.apache.org/jira/browse/HIVE-20394
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: denys kuzmenko
>Assignee: denys kuzmenko
>Priority: Major
> Attachments: HIVE-20394.1.patch, HIVE-20394.2.patch,
> HIVE-20394.3.patch, HIVE-20394.4.patch
>
>
> # Set proper cluster destroy order
> # Propagated proper HBaseTestContext
> # Ported downstream fixes (CDH-63695)
> # General clean up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[ https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-18453: - Attachment: HIVE-18453.07.patch > ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet > support > - > > Key: HIVE-18453 > URL: https://issues.apache.org/jira/browse/HIVE-18453 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch, > HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch, > HIVE-18453.06.patch, HIVE-18453.07.patch > > > The ACID table markers are currently done with TBLPROPERTIES which is > inherently fragile. > The "create transactional table" offers a way to standardize the syntax and > allows for future compatibility changes to support Parquet ACIDv2 tables > along with ORC tables. > The ACIDv2 design is format independent, with the ability to add new > vectorized input formats with no changes to the design. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18624) Parsing time is extremely high (~10 min) for queries with complex select expressions
[
https://issues.apache.org/jira/browse/HIVE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589079#comment-16589079
]
Hive QA commented on HIVE-18624:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936620/HIVE-18624.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14884 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13398/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13398/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13398/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936620 - PreCommit-HIVE-Build
> Parsing time is extremely high (~10 min) for queries with complex select
> expressions
>
>
> Key: HIVE-18624
> URL: https://issues.apache.org/jira/browse/HIVE-18624
> Project: Hive
> Issue Type: Bug
> Components: Hive, Parser
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Amruth S
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18624.01.patch, HIVE-18624.02.patch, thread_dump
>
>
> Explain of the same query takes
> 0.1 to 3 seconds in hive 2.1.0 &
> 10-15 min in hive 2.3.2 & latest master
> Sample expression below
> {code:java}
> EXPLAIN
> SELECT DISTINCT
> IF(lower('a') <= lower('a')
> ,'a'
> ,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
> ,'a'
> ,IF(if('a' = 'a', TRUE, FALSE) = 1
> ,'a'
> ,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
>and lower(if('a' = 'a','a','a')) <= lower('a'))
> OR ('a' like 'a' OR 'a' like 'a')
> OR 'a' in ('a','a')
> ,'a'
> ,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
> ,'a'
> ,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as
> string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp()
> <= unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) +
> 9*3600
> ,'a'
> ,If(lower('a') <= lower('a')
> and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
> ,'a'
> ,IF('a'=1 AND 'a'=1
> ,'a'
> ,IF('a' = 1 and COALESCE(cast('a' as int),0) = 0
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,If('a' = 'a' AND
> lower('a')>lower(if(lower('a')<1830,'a',cast(date_add('a',1) as timestamp)))
> ,'a'
> ,IF('a' = 1
> ,IF('a' in ('a', 'a') and ((unix_timestamp('a')-unix_timestamp('a')) / 60)
> > 30 and 'a' = 1
> ,'a', 'a')
> ,IF(if('a' = 'a', FALSE, TRUE ) = 1 AND 'a' IS NULL
> ,'a'
> ,IF('a' = 1 and 'a'>0
> , 'a'
> ,IF('a' = 1 AND 'a' ='a'
> ,'a'
> ,IF('a' is not null and 'a' is not null and 'a' > 'a'
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,If('a' = 1
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' = 1
> ,'a'
> ,IF('a' ='a' and 'a' ='a' and cast(unix_timestamp('a') as int) + 93600 <
> cast(unix_timestamp() as int)
> ,'a'
> ,IF('a' = 'a'
> ,'a'
> ,IF('a' = 'a' and 'a' in ('a','a','a')
> ,'a'
> ,IF('a' = 'a'
> ,'a','a'))
> )))
> AS test_comp_exp
> {code}
>
> Taking a look at [^thread_dump] shows a very large function stack getting
> created.
> Reverting HIVE-15578 (92f31d07aa988d4a460aac56e369bfa386361776) seem to speed
> up the parsing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589059#comment-16589059
]
Hui Huang edited comment on HIVE-20441 at 8/22/18 3:57 PM:
---
The patch return the new function object when call registerPermanentFunction
function.
was (Author: bigrey):
The patch return the new function object to client.
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Huang updated HIVE-20441:
-
Status: Patch Available (was: Open)
The patch return the new function object to client.
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Huang updated HIVE-20441:
-
Attachment: HIVE-20441.patch
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Attachments: HIVE-20441.patch
>
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
>
[jira] [Updated] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Huang updated HIVE-20441:
-
Description:
When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
started, the new created function from other clients or hiveserver2 will be
loaded from the metastore at the first time.
When the udf is used in where clause, we got a NPE like:
{code:java}
Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: NullPointerException null
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
SHOT]
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
T]
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
PSHOT]
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
PSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_77]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
~[hive-exec-2.
3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.
3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:229)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:176)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11613)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11568)
~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
at
[jira] [Assigned] (HIVE-20441) NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to true
[
https://issues.apache.org/jira/browse/HIVE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Huang reassigned HIVE-20441:
> NPE in ExprNodeGenericFuncDesc when hive.allow.udf.load.on.demand is set to
> true
> -
>
> Key: HIVE-20441
> URL: https://issues.apache.org/jira/browse/HIVE-20441
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
>
> When hive.allow.udf.load.on.demand is set to true and hiveserver2 has been
> started, the new created function from other clients or hiveserver2 will be
> loaded from the metastore at the first time.
> When the udf is used in where clause, we got a NPE like:
> {code:java}
> Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Error while compiling
> statement: FAILED: NullPointerException null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAP
> SHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHO
> T]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:542)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNA
> PSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57)
> ~[hive-service-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:236)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1104)
> ~[hive-exec-2.
> 3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.
> 3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> ~[hive-exec-2.3.4-SNAPSHOT.jar:2.3.4-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20351) GenericUDFNamedStruct should constant fold at compile time
[
https://issues.apache.org/jira/browse/HIVE-20351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589037#comment-16589037
]
Mykhailo Kysliuk commented on HIVE-20351:
-
[~gopalv]
Could you please review my patch?
> GenericUDFNamedStruct should constant fold at compile time
> --
>
> Key: HIVE-20351
> URL: https://issues.apache.org/jira/browse/HIVE-20351
> Project: Hive
> Issue Type: Bug
>Reporter: Mykhailo Kysliuk
>Assignee: Mykhailo Kysliuk
>Priority: Minor
> Attachments: HIVE-20351.1.patch, HIVE-20351.2.patch,
> HIVE-20351.3.patch, HIVE-20351.4.patch, HIVE-20351.5.patch
>
>
> Reproduced at hive-3.0.
> When we run hive query:
> {code:java}
> select named_struct('Total','Total') from test;
> {code}
> We could see the ERROR at hiveserver logs:
> {code:java}
> 2018-05-25T15:18:13,182 ERROR [main] optimizer.ConstantPropagateProcFactory:
> Unable to evaluate
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNamedStruct@a0bf272. Return
> value unrecoginizable.
> {code}
> This error is harmless because all results are correct. But named_struct
> constant values should be processed correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18624) Parsing time is extremely high (~10 min) for queries with complex select expressions
[
https://issues.apache.org/jira/browse/HIVE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589034#comment-16589034
]
Hive QA commented on HIVE-18624:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 3 new + 16 unchanged - 2 fixed
= 19 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13398/dev-support/hive-personality.sh
|
| git revision | master / 09c6360 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13398/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13398/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Parsing time is extremely high (~10 min) for queries with complex select
> expressions
>
>
> Key: HIVE-18624
> URL: https://issues.apache.org/jira/browse/HIVE-18624
> Project: Hive
> Issue Type: Bug
> Components: Hive, Parser
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Amruth S
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18624.01.patch, HIVE-18624.02.patch, thread_dump
>
>
> Explain of the same query takes
> 0.1 to 3 seconds in hive 2.1.0 &
> 10-15 min in hive 2.3.2 & latest master
> Sample expression below
> {code:java}
> EXPLAIN
> SELECT DISTINCT
> IF(lower('a') <= lower('a')
> ,'a'
> ,IF(('a' IS NULL AND from_unixtime(UNIX_TIMESTAMP()) <= 'a')
> ,'a'
> ,IF(if('a' = 'a', TRUE, FALSE) = 1
> ,'a'
> ,IF(('a' = 1 and lower('a') NOT IN ('a', 'a')
>and lower(if('a' = 'a','a','a')) <= lower('a'))
> OR ('a' like 'a' OR 'a' like 'a')
> OR 'a' in ('a','a')
> ,'a'
> ,IF(if(lower('a') in ('a', 'a') and 'a'='a', TRUE, FALSE) = 1
> ,'a'
> ,IF('a'='a' and unix_timestamp(if('a' = 'a',cast('a' as
> string),coalesce('a',cast('a' as string),from_unixtime(unix_timestamp()
> <= unix_timestamp(concat_ws('a',cast(lower('a') as string),'00:00:00')) +
> 9*3600
> ,'a'
> ,If(lower('a') <= lower('a')
> and if(lower('a') in ('a', 'a') and 'a'<>'a', TRUE, FALSE) <> 1
> ,'a'
> ,IF('a'=1 AND 'a'=1
[jira] [Updated] (HIVE-20374) Write Hive version information to Parquet footer
[ https://issues.apache.org/jira/browse/HIVE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20374: - Attachment: HIVE-20374.03.patch > Write Hive version information to Parquet footer > > > Key: HIVE-20374 > URL: https://issues.apache.org/jira/browse/HIVE-20374 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Ivanfi >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20374.01.patch, HIVE-20374.02.patch, > HIVE-20374.03.patch, HIVE-20374.patch > > > PARQUET-352 added support for the "writer.model.name" property in the Parquet > metadata to identify the object model (application) that wrote the file. > The easiest way to write this property is by overriding getName() of > org.apache.parquet.hadoop.api.WriteSupport. In Hive, this would mean adding > getName() to the > org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20374) Write Hive version information to Parquet footer
[
https://issues.apache.org/jira/browse/HIVE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588992#comment-16588992
]
Hive QA commented on HIVE-20374:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936611/HIVE-20374.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14881 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=116)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13397/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13397/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13397/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936611 - PreCommit-HIVE-Build
> Write Hive version information to Parquet footer
>
>
> Key: HIVE-20374
> URL: https://issues.apache.org/jira/browse/HIVE-20374
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Ivanfi
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20374.01.patch, HIVE-20374.02.patch,
> HIVE-20374.patch
>
>
> PARQUET-352 added support for the "writer.model.name" property in the Parquet
> metadata to identify the object model (application) that wrote the file.
> The easiest way to write this property is by overriding getName() of
> org.apache.parquet.hadoop.api.WriteSupport. In Hive, this would mean adding
> getName() to the
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18565) Set some timeout to all CliDriver tests
[ https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18565: Attachment: HIVE-18565.06.patch > Set some timeout to all CliDriver tests > --- > > Key: HIVE-18565 > URL: https://issues.apache.org/jira/browse/HIVE-18565 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18565.01.patch, HIVE-18565.02.patch, > HIVE-18565.03.patch, HIVE-18565.04.patch, HIVE-18565.05.patch, > HIVE-18565.05.patch, HIVE-18565.06.patch > > > if a testcase runs into an infinite loop or something ; it should still fail > instead of causing timeout at the ptest executor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20374) Write Hive version information to Parquet footer
[
https://issues.apache.org/jira/browse/HIVE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588908#comment-16588908
]
Hive QA commented on HIVE-20374:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13397/dev-support/hive-personality.sh
|
| git revision | master / 09c6360 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13397/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Write Hive version information to Parquet footer
>
>
> Key: HIVE-20374
> URL: https://issues.apache.org/jira/browse/HIVE-20374
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Ivanfi
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20374.01.patch, HIVE-20374.02.patch,
> HIVE-20374.patch
>
>
> PARQUET-352 added support for the "writer.model.name" property in the Parquet
> metadata to identify the object model (application) that wrote the file.
> The easiest way to write this property is by overriding getName() of
> org.apache.parquet.hadoop.api.WriteSupport. In Hive, this would mean adding
> getName() to the
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20396) Test HS2 open_connection metrics
[ https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20396: - Attachment: HIVE-20396.06.patch > Test HS2 open_connection metrics > > > Key: HIVE-20396 > URL: https://issues.apache.org/jira/browse/HIVE-20396 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20396.01.patch, HIVE-20396.02.patch, > HIVE-20396.03.patch, HIVE-20396.04.patch, HIVE-20396.05.patch, > HIVE-20396.06.patch, HIVE-20396.patch > > > HiveServer2 is emitting metrics _default.General.open_connections_ in both > binary and http mode. These metrics should be tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20396) Test HS2 open_connection metrics
[
https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588868#comment-16588868
]
Hive QA commented on HIVE-20396:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936597/HIVE-20396.05.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14883 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13396/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13396/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936597 - PreCommit-HIVE-Build
> Test HS2 open_connection metrics
>
>
> Key: HIVE-20396
> URL: https://issues.apache.org/jira/browse/HIVE-20396
> Project: Hive
> Issue Type: Test
> Components: HiveServer2
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20396.01.patch, HIVE-20396.02.patch,
> HIVE-20396.03.patch, HIVE-20396.04.patch, HIVE-20396.05.patch,
> HIVE-20396.patch
>
>
> HiveServer2 is emitting metrics _default.General.open_connections_ in both
> binary and http mode. These metrics should be tested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.03.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.02.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20396) Test HS2 open_connection metrics
[
https://issues.apache.org/jira/browse/HIVE-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588816#comment-16588816
]
Hive QA commented on HIVE-20396:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13396/dev-support/hive-personality.sh
|
| git revision | master / 09c6360 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13396/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: itests/hive-unit U: itests/hive-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13396/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Test HS2 open_connection metrics
>
>
> Key: HIVE-20396
> URL: https://issues.apache.org/jira/browse/HIVE-20396
> Project: Hive
> Issue Type: Test
> Components: HiveServer2
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-20396.01.patch, HIVE-20396.02.patch,
> HIVE-20396.03.patch, HIVE-20396.04.patch, HIVE-20396.05.patch,
> HIVE-20396.patch
>
>
> HiveServer2 is emitting metrics _default.General.open_connections_ in both
> binary and http mode. These metrics should be tested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20439) Use the inflated memory limit during join selection for llap
[ https://issues.apache.org/jira/browse/HIVE-20439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20439: Status: Patch Available (was: Open) > Use the inflated memory limit during join selection for llap > > > Key: HIVE-20439 > URL: https://issues.apache.org/jira/browse/HIVE-20439 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20439.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20433) Implicit String to Timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588795#comment-16588795
]
Hive QA commented on HIVE-20433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936596/HIVE-20433.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13395/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13395/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13395/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12936596/HIVE-20433.4.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936596 - PreCommit-HIVE-Build
> Implicit String to Timestamp conversion is slow
> ---
>
> Key: HIVE-20433
> URL: https://issues.apache.org/jira/browse/HIVE-20433
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch,
> HIVE-20433.3.patch, HIVE-20433.4.patch
>
>
> getTimestampFromString() is slow at casting dates. It throws twice before
> date conversion can happen.
>
> cc [~gopalv] [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20433) Implicit String to Timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588793#comment-16588793
]
Hive QA commented on HIVE-20433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936596/HIVE-20433.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14881 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_from_utc_timestamp]
(batchId=89)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13394/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13394/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13394/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936596 - PreCommit-HIVE-Build
> Implicit String to Timestamp conversion is slow
> ---
>
> Key: HIVE-20433
> URL: https://issues.apache.org/jira/browse/HIVE-20433
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch,
> HIVE-20433.3.patch, HIVE-20433.4.patch
>
>
> getTimestampFromString() is slow at casting dates. It throws twice before
> date conversion can happen.
>
> cc [~gopalv] [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Status: Patch Available (was: Open) > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.01.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20439) Use the inflated memory limit during join selection for llap
[ https://issues.apache.org/jira/browse/HIVE-20439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20439: Attachment: HIVE-20439.01.patch > Use the inflated memory limit during join selection for llap > > > Key: HIVE-20439 > URL: https://issues.apache.org/jira/browse/HIVE-20439 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20439.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-20440: -- > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14162) Allow disabling of long running job on Hive On Spark On YARN
[ https://issues.apache.org/jira/browse/HIVE-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588781#comment-16588781 ] Sahil Takiar commented on HIVE-14162: - [~szita], [~asinkovits] could you take a look? > Allow disabling of long running job on Hive On Spark On YARN > > > Key: HIVE-14162 > URL: https://issues.apache.org/jira/browse/HIVE-14162 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Thomas Scott >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14162.1.patch, HIVE-14162.2.patch, > HIVE-14162.3.patch, HIVE-14162.4.patch, HIVE-14162.5.patch, > HIVE-14162.6.patch, HIVE-14162.7.patch > > > Hive On Spark launches a long running process on the first query to handle > all queries for that user session. In some use cases this is not desired, for > instance when using Hue with large intervals between query executions. > Could we have a property that would cause long running spark jobs to be > terminated after each query execution and started again for the next one? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20433) Implicit String to Timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588760#comment-16588760
]
Hive QA commented on HIVE-20433:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} serde in master has 195 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 2 new + 190 unchanged - 0
fixed = 192 total (was 190) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13394/dev-support/hive-personality.sh
|
| git revision | master / 09c6360 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13394/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13394/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implicit String to Timestamp conversion is slow
> ---
>
> Key: HIVE-20433
> URL: https://issues.apache.org/jira/browse/HIVE-20433
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20433.1.patch, HIVE-20433.2.patch,
> HIVE-20433.3.patch, HIVE-20433.4.patch
>
>
> getTimestampFromString() is slow at casting dates. It throws twice before
> date conversion can happen.
>
> cc [~gopalv] [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20439) Use the inflated memory limit during join selection for llap
[ https://issues.apache.org/jira/browse/HIVE-20439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20439: --- > Use the inflated memory limit during join selection for llap > > > Key: HIVE-20439 > URL: https://issues.apache.org/jira/browse/HIVE-20439 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20437) Handle schema evolution from float, double and decimal
[
https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588729#comment-16588729
]
Hive QA commented on HIVE-20437:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936565/HIVE-20437.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14887 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedListColumnReader.testListReadEqualOneBatch
(batchId=287)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedListColumnReader.testListReadLessOneBatch
(batchId=287)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedListColumnReader.testListReadMoreOneBatch
(batchId=287)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedMapColumnReader.testMapReadEqualOneBatch
(batchId=288)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedMapColumnReader.testMapReadLessOneBatch
(batchId=288)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedMapColumnReader.testMapReadMoreOneBatch
(batchId=288)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testDateCharTypes
(batchId=206)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimal
(batchId=206)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimalX
(batchId=206)
org.apache.hive.hcatalog.pig.TestParquetHCatStorer.testWriteDecimalXY
(batchId=206)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13393/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13393/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12936565 - PreCommit-HIVE-Build
> Handle schema evolution from float, double and decimal
> --
>
> Key: HIVE-20437
> URL: https://issues.apache.org/jira/browse/HIVE-20437
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20437.1.patch
>
>
> When data created as float, double or decimal in parquet format is read back
> using some other type, errors are seen. Parquet should behave just like any
> other format. If the value is valid for the new type, data is retuned
> otherwise null has to be returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20151) External table: exception while storing stats
[
https://issues.apache.org/jira/browse/HIVE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-20151.
-
Resolution: Duplicate
> External table: exception while storing stats
> -
>
> Key: HIVE-20151
> URL: https://issues.apache.org/jira/browse/HIVE-20151
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Statistics
>Reporter: Zoltan Haindrich
>Priority: Major
>
> {code}
> create external table e3(a integer,b string,c double);
> -- goes well
> insert into e3 values(1,'2',3);
> -- takes a while:
> insert into e3 values(1,'2',3);
> -- after 2 minutes
> --
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.StatsTask
> INFO : Completed executing
> command(queryId=hive_20180712120342_6893e234-44a0-4e48-8320-f1699557bae3);
> Time taken: 125.276 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)
> {code}
> exception in metastore logs:
> {code}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
> org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1084)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7514)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_161]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at com.sun.proxy.$Proxy34.set_aggr_stats_for(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:17017)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:17001)
> ~[hive-exec-3.1.0.3.0.0.0-1632.jar:3.1.0.3.0.0.0-1632]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17084: Attachment: HIVE-170884.7.patch > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.1.patch, HIVE-170884.4.patch, > HIVE-170884.5.patch, HIVE-170884.7.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17084: Status: Patch Available (was: Open) > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.1.patch, HIVE-170884.4.patch, > HIVE-170884.5.patch, HIVE-170884.7.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17084: Status: Open (was: Patch Available) > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.1.patch, HIVE-170884.4.patch, > HIVE-170884.5.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20437) Handle schema evolution from float, double and decimal
[
https://issues.apache.org/jira/browse/HIVE-20437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588719#comment-16588719
]
Hive QA commented on HIVE-20437:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
32s{color} | {color:red} ql generated 1 new + 2308 unchanged - 0 fixed = 2309
total (was 2308) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Load of known null value in
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8.getConverter(PrimitiveType,
int, ConverterParent, TypeInfo) At ETypeConverter.java:in
org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8.getConverter(PrimitiveType,
int, ConverterParent, TypeInfo) At ETypeConverter.java:[line 586] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13393/dev-support/hive-personality.sh
|
| git revision | master / 25e747f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13393/yetus/new-findbugs-ql.html
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13393/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Handle schema evolution from float, double and decimal
> --
>
> Key: HIVE-20437
> URL: https://issues.apache.org/jira/browse/HIVE-20437
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-20437.1.patch
>
>
> When data created as float, double or decimal in parquet format is read back
> using some other type, errors are seen. Parquet should behave just like any
> other format. If the value is valid for the new type, data is retuned
> otherwise null has to be returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20421) Illegal character entity '\b' in hive-default.xml.template
[
https://issues.apache.org/jira/browse/HIVE-20421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20421:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Igor!
> Illegal character entity '\b' in hive-default.xml.template
> --
>
> Key: HIVE-20421
> URL: https://issues.apache.org/jira/browse/HIVE-20421
> Project: Hive
> Issue Type: Bug
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20421.01.patch
>
>
> Illegal {{\b}} element in {{hive-site.xml. }}
> {code}
> TXN_OVERWRITE_X_LOCK("hive.txn.xlock.iow", true,
> "Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire
> Exclusive locks for\b" +
> "transactional tables. This ensures that inserts (w/o overwrite)
> running concurrently\n" +
> "are not hidden by the INSERT OVERWRITE."),
> {code}
> Seems like just typo of {{\n}}.
> Stacktrace
> {code}
> Exception in thread "main" java.lang.RuntimeException:
> com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion
> character (code 0x8
> at [row,col,system-id]:
> [3278,96,"file:/usr/local/apache-hive/conf/hive-site.xml"]
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
> at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5059)
> at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5132)
> at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5219)
> at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:5167)
> at
> org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:138)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
