[jira] [Commented] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548805#comment-16548805
]
Hive QA commented on HIVE-15131:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12686/dev-support/hive-personality.sh
|
| git revision | master / d5c589b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12686/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch, HIVE-15131.4.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20164) Murmur Hash : Make sure CTAS and IAS use correct bucketing version
[ https://issues.apache.org/jira/browse/HIVE-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20164: -- Attachment: HIVE-20164.4.patch > Murmur Hash : Make sure CTAS and IAS use correct bucketing version > -- > > Key: HIVE-20164 > URL: https://issues.apache.org/jira/browse/HIVE-20164 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20164.1.patch, HIVE-20164.2.patch, > HIVE-20164.3.patch, HIVE-20164.4.patch > > > With the migration to Murmur hash, CTAS and IAS from old table version to new > table version does not work as intended and data is hashed using old hash > logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[
https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548791#comment-16548791
]
Hive QA commented on HIVE-20202:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932173/HIVE-20202.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12685/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12685/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12685/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932173 - PreCommit-HIVE-Build
> Add profiler endpoint to HS2 and LLAP
> -
>
> Key: HIVE-20202
> URL: https://issues.apache.org/jira/browse/HIVE-20202
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch,
> HIVE-20202.3.patch
>
>
> Add a web endpoint for profiling based on async-profiler. This servlet should
> be added to httpserver so that HS2 and LLAP daemons can output flamegraphs
> when their /prof endpoint is hit. Since this will be based on
> [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock
> contentions, HW counters etc. will also be supported in addition to cpu
> profiling. In most cases the profiling overhead is pretty low and is safe to
> run on production. More analysis on CPU and memory overhead here
> [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and
> [https://github.com/jvm-profiling-tools/async-profiler/issues/131]
>
> For the impatient, here is the usage doc and the sample output
> [https://github.com/prasanthj/nightswatch/blob/master/README.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[
https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548786#comment-16548786
]
Hive QA commented on HIVE-20202:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} common: The patch generated 5 new + 19 unchanged - 0
fixed = 24 total (was 19) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} root: The patch generated 5 new + 19 unchanged - 0
fixed = 24 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12685/dev-support/hive-personality.sh
|
| git revision | master / d5c589b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12685/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12685/yetus/diff-checkstyle-root.txt
|
| modules | C: common . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12685/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add profiler endpoint to HS2 and LLAP
> -
>
> Key: HIVE-20202
> URL: https://issues.apache.org/jira/browse/HIVE-20202
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch,
> HIVE-20202.3.patch
>
>
> Add a web endpoint for profiling based on async-profiler. This servlet should
> be added to httpserver so that HS2 and LLAP daemons can output flamegraphs
> when their /prof endpoint is hit. Since this will be based on
> [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock
> contentions, HW counters etc. will also be supported in addition to cpu
> profiling. In most cases the profiling overhead is pretty low and is safe to
> run on
[jira] [Commented] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548782#comment-16548782 ] Vineet Garg commented on HIVE-20208: [~EricWohlstadter] If you were already looking into it feel free to assign this back to yourself. > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Assignee: Vineet Garg >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20208: -- Assignee: Vineet Garg (was: Eric Wohlstadter) > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Assignee: Vineet Garg >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548756#comment-16548756
]
Hive QA commented on HIVE-19166:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932125/HIVE-19166.16.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14663 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12684/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12684/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12684/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932125 - PreCommit-HIVE-Build
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch,
> HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch,
> HIVE-19166.16.patch, HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xulongfetion reassigned HIVE-20208: --- Assignee: Eric Wohlstadter (was: Hive QA) > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Assignee: Eric Wohlstadter >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xulongfetion updated HIVE-20208: Attachment: (was: order_by_position_alias.patch) > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Assignee: Hive QA >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xulongfetion updated HIVE-20208: Assignee: Hive QA Attachment: order_by_position_alias.patch Status: Patch Available (was: Open) > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Assignee: Hive QA >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch, > order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20208) order by position alias do not work
[ https://issues.apache.org/jira/browse/HIVE-20208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xulongfetion updated HIVE-20208: Flags: Patch Labels: patch-available (was: ) Priority: Blocker (was: Major) > order by position alias do not work > --- > > Key: HIVE-20208 > URL: https://issues.apache.org/jira/browse/HIVE-20208 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: xulongfetion >Priority: Blocker > Labels: patch-available > Attachments: order_by_position_alias.patch > > > the order by position alias do not work if there's more than one position > alias. For query like > "select a, b, c from table1 order by 1, 2;", hive will report column not > found error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19360) CBO: Add an "optimizedSQL" to QueryPlan object
[
https://issues.apache.org/jira/browse/HIVE-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19360:
---
Attachment: HIVE-19360.8.patch
> CBO: Add an "optimizedSQL" to QueryPlan object
> ---
>
> Key: HIVE-19360
> URL: https://issues.apache.org/jira/browse/HIVE-19360
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Diagnosability
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19360.1.patch, HIVE-19360.2.patch,
> HIVE-19360.3.patch, HIVE-19360.4.patch, HIVE-19360.5.patch,
> HIVE-19360.6.patch, HIVE-19360.7.patch, HIVE-19360.8.patch
>
>
> Calcite RelNodes can be converted back into SQL (as the new JDBC storage
> handler does), which allows Hive to print out the post CBO plan as a SQL
> query instead of having to guess the join orders from the subsequent Tez plan.
> The query generated might not be always valid SQL at this point, but is a
> world ahead of DAG plans in readability.
> Eg. tpc-ds Query4 CTEs gets expanded to
> {code}
> SELECT t16.$f3 customer_preferred_cust_flag
> FROM
> (SELECT t0.c_customer_id $f0,
>SUM((t2.ws_ext_list_price -
> t2.ws_ext_wholesale_cost - t2.ws_ext_discount_amt + t2.ws_ext_sales_price) /
> CAST(2 AS DECIMAL(10, 0))) $f8
>FROM
> (SELECT c_customer_sk,
> c_customer_id,
> c_first_name,
> c_last_name,
> c_preferred_cust_flag,
> c_birth_country,
> c_login,
> c_email_address
> FROM default.customer
> WHERE c_customer_sk IS NOT NULL
> AND c_customer_id IS NOT NULL) t0
>INNER JOIN (
> (SELECT ws_sold_date_sk,
> ws_bill_customer_sk,
> ws_ext_discount_amt,
> ws_ext_sales_price,
> ws_ext_wholesale_cost,
> ws_ext_list_price
> FROM default.web_sales
> WHERE ws_bill_customer_sk IS NOT NULL
> AND ws_sold_date_sk IS NOT NULL) t2
>INNER JOIN
> (SELECT d_date_sk,
> CAST(2002 AS INTEGER) d_year
> FROM default.date_dim
> WHERE d_year = 2002
> AND d_date_sk IS NOT NULL) t4 ON t2.ws_sold_date_sk =
> t4.d_date_sk) ON t0.c_customer_sk = t2.ws_bill_customer_sk
>GROUP BY t0.c_customer_id,
> t0.c_first_name,
> t0.c_last_name,
> t0.c_preferred_cust_flag,
> t0.c_birth_country,
> t0.c_login,
> t0.c_email_address) t7
> INNER JOIN (
> (SELECT t9.c_customer_id $f0,
>t9.c_preferred_cust_flag $f3,
>
> SUM((t11.ss_ext_list_price - t11.ss_ext_wholesale_cost -
> t11.ss_ext_discount_amt + t11.ss_ext_sales_price) / CAST(2 AS DECIMAL(10,
> 0))) $f8
>FROM
> (SELECT c_customer_sk,
> c_customer_id,
> c_first_name,
> c_last_name,
> c_preferred_cust_flag,
> c_birth_country,
> c_login,
> c_email_address
> FROM default.customer
> WHERE c_customer_sk IS NOT NULL
> AND c_customer_id IS NOT NULL) t9
>INNER JOIN (
> (SELECT ss_sold_date_sk,
> ss_customer_sk,
> ss_ext_discount_amt,
> ss_ext_sales_price,
> ss_ext_wholesale_cost,
> ss_ext_list_price
> FROM default.store_sales
> WHERE ss_customer_sk IS NOT NULL
> AND ss_sold_date_sk IS NOT NULL) t11
>INNER JOIN
> (SELECT d_date_sk,
> CAST(2002 AS INTEGER) d_year
> FROM default.date_dim
> WHERE d_year = 2002
> AND d_date_sk IS NOT NULL) t13 ON
> t11.ss_sold_date_sk = t13.d_date_sk) ON t9.c_customer_sk = t11.ss_customer_sk
>GROUP BY t9.c_customer_id,
> t9.c_first_name,
> t9.c_last_name,
> t9.c_preferred_cust_flag,
>
[jira] [Commented] (HIVE-19166) TestMiniLlapLocalCliDriver sysdb failure
[
https://issues.apache.org/jira/browse/HIVE-19166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548726#comment-16548726
]
Hive QA commented on HIVE-19166:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12684/dev-support/hive-personality.sh
|
| git revision | master / d5c589b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: metastore ql itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12684/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestMiniLlapLocalCliDriver sysdb failure
>
>
> Key: HIVE-19166
> URL: https://issues.apache.org/jira/browse/HIVE-19166
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19166.04.patch, HIVE-19166.05.patch,
> HIVE-19166.06.patch, HIVE-19166.09.patch, HIVE-19166.1.patch,
> HIVE-19166.10.patch, HIVE-19166.11.patch, HIVE-19166.12.patch,
> HIVE-19166.13.patch, HIVE-19166.14.patch, HIVE-19166.15.patch,
> HIVE-19166.16.patch, HIVE-19166.2.patch, HIVE-19166.3.patch
>
>
> Broken by HIVE-18715
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20198) Constant time table drops/renames
[
https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548724#comment-16548724
]
Vihang Karajgaonkar edited comment on HIVE-20198 at 7/19/18 3:00 AM:
-
{{TBLS.TBL_ID}} is internally managed by datanucleus and not exposed to the
Thrift Table object. Its value is dependent on the backing database according
to datanucleus documentation (although I have almost always seen it as a
monotonically increasing number). Its value is guaranteed to be unique so I
think we can potentially use it. But in my opinion there is still value to
expose such a id which is controlled by metastore at the thrift level. For
instance such ids can be used to identify versions of the objects to provide
optimistic concurrency control model instead of the lock based concurrency
model we have currently. When a table object which is altered metastore can
change uuid but different currently the TBL_ID which managed by datanucleus
does not change as far as I understand.
was (Author: vihangk1):
{{TBLS.TBL_ID}} is internally managed by datanucleus and not exposed to the
Thrift Table object. Its value is dependent on the backing database according
to datanucleus documentation (although I have almost always seen it as a
monotonically increasing number). Its value is guaranteed to be unique so I
think we can potentially use it. But in my opinion there is still value to
expose such a id which is controlled by metastore at the thrift level. For
instance such ids can be used to identify versions of the objects to provide
optimistic concurrency control model instead of the lock based concurrency
model we have currently. A table object which is altered can have a different
uuid but not different currently the TBL_ID does not change.
> Constant time table drops/renames
> -
>
> Key: HIVE-20198
> URL: https://issues.apache.org/jira/browse/HIVE-20198
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> Currently table drops and table renames have O(P) performance (where P is the
> number of partitions). When a managed table is deleted, the implementation
> deletes table metadata and then deletes all partitions in HDFS. HDFS
> operations are optimized and only do a sequential deletes for partitions
> outside of table prefix. This operation is O(P)where Pis the number of
> partitions.
> Table rename goes through the list of partitions and modifies table name (and
> potentially db name) in each partition. It also modifies each partition
> location to match the new db/table name and renames directories (which is a
> non-atomic and slow operation on S3). This is O(P) operation where P is the
> number of partitions.
> Basic idea is to do the following:
> # Assign unique ID to each table
> # Create directory name based on unique ID rather then the name
> # Table rename then becomes metadata-only operation - there is no need to
> change any location information.
> # Table drop can become an asynchronous operation where the table is marked
> as "deleted". Subsequent public metadata APIs should skip such tables. A
> background cleaner thread may then go and clean up directories.
> Since the table location is unique for each table, new tables will not reuse
> existing locations. This change isn't compatible with the current behavior
> where there is an assumption that table location is based on table name. We
> can get around this by providing "opt-in" mechanism - special table property
> that tells that the table can have such new behavior, so the improvement will
> initially work for new tables created with this feature enabled. We may later
> provide some tool to convert existing tables to the new scheme.
> One complication is there in case where impersonation is enabled - the FS
> operations should be performed using client UGI rather then server's, so the
> cleaner thread should be able to use client UGIs.
> Initially we can punt on this and do standard table drops when impersonation
> is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20198) Constant time table drops/renames
[
https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548724#comment-16548724
]
Vihang Karajgaonkar commented on HIVE-20198:
{{TBLS.TBL_ID}} is internally managed by datanucleus and not exposed to the
Thrift Table object. Its value is dependent on the backing database according
to datanucleus documentation (although I have almost always seen it as a
monotonically increasing number). Its value is guaranteed to be unique so I
think we can potentially use it. But in my opinion there is still value to
expose such a id which is controlled by metastore at the thrift level. For
instance such ids can be used to identify versions of the objects to provide
optimistic concurrency control model instead of the lock based concurrency
model we have currently. A table object which is altered can have a different
uuid but not different currently the TBL_ID does not change.
> Constant time table drops/renames
> -
>
> Key: HIVE-20198
> URL: https://issues.apache.org/jira/browse/HIVE-20198
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> Currently table drops and table renames have O(P) performance (where P is the
> number of partitions). When a managed table is deleted, the implementation
> deletes table metadata and then deletes all partitions in HDFS. HDFS
> operations are optimized and only do a sequential deletes for partitions
> outside of table prefix. This operation is O(P)where Pis the number of
> partitions.
> Table rename goes through the list of partitions and modifies table name (and
> potentially db name) in each partition. It also modifies each partition
> location to match the new db/table name and renames directories (which is a
> non-atomic and slow operation on S3). This is O(P) operation where P is the
> number of partitions.
> Basic idea is to do the following:
> # Assign unique ID to each table
> # Create directory name based on unique ID rather then the name
> # Table rename then becomes metadata-only operation - there is no need to
> change any location information.
> # Table drop can become an asynchronous operation where the table is marked
> as "deleted". Subsequent public metadata APIs should skip such tables. A
> background cleaner thread may then go and clean up directories.
> Since the table location is unique for each table, new tables will not reuse
> existing locations. This change isn't compatible with the current behavior
> where there is an assumption that table location is based on table name. We
> can get around this by providing "opt-in" mechanism - special table property
> that tells that the table can have such new behavior, so the improvement will
> initially work for new tables created with this feature enabled. We may later
> provide some tool to convert existing tables to the new scheme.
> One complication is there in case where impersonation is enabled - the FS
> operations should be performed using client UGI rather then server's, so the
> cleaner thread should be able to use client UGIs.
> Initially we can punt on this and do standard table drops when impersonation
> is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[
https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548704#comment-16548704
]
Sahil Takiar commented on HIVE-20032:
-
[~lirui] attached an updated batch that preserves the Kryo shading. It moves
the new serializer to the {{kryo-registrator}} module and uses reflection to
instantiate the class.
> Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
> -
>
> Key: HIVE-20032
> URL: https://issues.apache.org/jira/browse/HIVE-20032
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch,
> HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch
>
>
> Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles,
> then we don't need to serialize the hashCode when shuffling data in HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Attachment: HIVE-20032.5.patch > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch, > HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19915) master-txnstats branch - support CachedStore
[ https://issues.apache.org/jira/browse/HIVE-19915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548688#comment-16548688 ] Sergey Shelukhin commented on HIVE-19915: - [~steveyeom2017] do you have any update here, what is the current state? I can take a look > master-txnstats branch - support CachedStore > > > Key: HIVE-19915 > URL: https://issues.apache.org/jira/browse/HIVE-19915 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Steve Yeom >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20047: Attachment: HIVE-20047.patch > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20047: --- Assignee: Sergey Shelukhin > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548687#comment-16548687 ] Sergey Shelukhin commented on HIVE-20047: - Updated to remove the API arg. Let's see what tests fail on top of the existing failures in the branch. > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20047: Status: Patch Available (was: Reopened) > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548682#comment-16548682
]
Hive QA commented on HIVE-20203:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932119/HIVE-20203.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12683/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12683/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12683/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932119 - PreCommit-HIVE-Build
> Arrow SerDe leaks a DirectByteBuffer
>
>
> Key: HIVE-20203
> URL: https://issues.apache.org/jira/browse/HIVE-20203
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: HIVE-20203.1.patch
>
>
> ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task
> that uses the serde.
> The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.
> This buffer is never closed and leaks about 1K of physical memory for each
> task.
> This patch does three things:
> # Ensure the buffer is closed when the RecordWriter for the task is closed.
> # Adds per-task memory accounting by assigning a ChildAllocator to each task
> from the RootAllocator.
> # Enforces that the ChildAllocator for a task has released all memory
> assigned to it, when the task is completed.
> The patch assumes that close() is always called on the RecordWriter when a
> task is finished (even if there is a failure during task execution).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548668#comment-16548668
]
Hive QA commented on HIVE-20203:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 303 unchanged - 0
fixed = 305 total (was 303) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12683/dev-support/hive-personality.sh
|
| git revision | master / d5c589b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12683/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12683/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Arrow SerDe leaks a DirectByteBuffer
>
>
> Key: HIVE-20203
> URL: https://issues.apache.org/jira/browse/HIVE-20203
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: HIVE-20203.1.patch
>
>
> ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task
> that uses the serde.
> The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.
> This buffer is never closed and leaks about 1K of physical memory for each
> task.
> This patch does three things:
> # Ensure the buffer is closed when the RecordWriter for the task is closed.
> # Adds per-task memory accounting by assigning a ChildAllocator to each task
> from the RootAllocator.
> # Enforces that the ChildAllocator for a task has released all memory
> assigned to it, when the task is completed.
> The patch assumes that close() is always called on the RecordWriter when a
> task is finished (even
[jira] [Commented] (HIVE-20164) Murmur Hash : Make sure CTAS and IAS use correct bucketing version
[
https://issues.apache.org/jira/browse/HIVE-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548638#comment-16548638
]
Hive QA commented on HIVE-20164:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932165/HIVE-20164.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14663 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[murmur_hash_migration]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12682/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12682/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12682/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932165 - PreCommit-HIVE-Build
> Murmur Hash : Make sure CTAS and IAS use correct bucketing version
> --
>
> Key: HIVE-20164
> URL: https://issues.apache.org/jira/browse/HIVE-20164
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20164.1.patch, HIVE-20164.2.patch,
> HIVE-20164.3.patch
>
>
> With the migration to Murmur hash, CTAS and IAS from old table version to new
> table version does not work as intended and data is hashed using old hash
> logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[ https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548636#comment-16548636 ] Prasanth Jayachandran commented on HIVE-20202: -- Addressed review comments. > Add profiler endpoint to HS2 and LLAP > - > > Key: HIVE-20202 > URL: https://issues.apache.org/jira/browse/HIVE-20202 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch, > HIVE-20202.3.patch > > > Add a web endpoint for profiling based on async-profiler. This servlet should > be added to httpserver so that HS2 and LLAP daemons can output flamegraphs > when their /prof endpoint is hit. Since this will be based on > [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock > contentions, HW counters etc. will also be supported in addition to cpu > profiling. In most cases the profiling overhead is pretty low and is safe to > run on production. More analysis on CPU and memory overhead here > [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and > [https://github.com/jvm-profiling-tools/async-profiler/issues/131] > > For the impatient, here is the usage doc and the sample output > [https://github.com/prasanthj/nightswatch/blob/master/README.md] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[ https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20202: - Attachment: HIVE-20202.3.patch > Add profiler endpoint to HS2 and LLAP > - > > Key: HIVE-20202 > URL: https://issues.apache.org/jira/browse/HIVE-20202 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch, > HIVE-20202.3.patch > > > Add a web endpoint for profiling based on async-profiler. This servlet should > be added to httpserver so that HS2 and LLAP daemons can output flamegraphs > when their /prof endpoint is hit. Since this will be based on > [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock > contentions, HW counters etc. will also be supported in addition to cpu > profiling. In most cases the profiling overhead is pretty low and is safe to > run on production. More analysis on CPU and memory overhead here > [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and > [https://github.com/jvm-profiling-tools/async-profiler/issues/131] > > For the impatient, here is the usage doc and the sample output > [https://github.com/prasanthj/nightswatch/blob/master/README.md] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations
[
https://issues.apache.org/jira/browse/HIVE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548630#comment-16548630
]
Jason Dere commented on HIVE-20204:
---
RB at https://reviews.apache.org/r/67970/
> Type conversion during IN () comparisons is using different rules from other
> comparison operations
> --
>
> Key: HIVE-20204
> URL: https://issues.apache.org/jira/browse/HIVE-20204
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20204.1.patch
>
>
> Noticed this while looking at HIVE-20082.
> The type conversion done during GenericUDFIn (via
> ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(),
> whereas the other comparison operators (=, <, >, <=, >=) use
> FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN
> ('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1'
> would compare the values as doubles. This makes a difference for HIVE-20082
> since it is related to changing the 0-padding during decimal-to-string
> conversions.
> cc [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations
[
https://issues.apache.org/jira/browse/HIVE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20204:
--
Status: Patch Available (was: Open)
> Type conversion during IN () comparisons is using different rules from other
> comparison operations
> --
>
> Key: HIVE-20204
> URL: https://issues.apache.org/jira/browse/HIVE-20204
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20204.1.patch
>
>
> Noticed this while looking at HIVE-20082.
> The type conversion done during GenericUDFIn (via
> ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(),
> whereas the other comparison operators (=, <, >, <=, >=) use
> FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN
> ('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1'
> would compare the values as doubles. This makes a difference for HIVE-20082
> since it is related to changing the 0-padding during decimal-to-string
> conversions.
> cc [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations
[
https://issues.apache.org/jira/browse/HIVE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20204:
--
Attachment: HIVE-20204.1.patch
> Type conversion during IN () comparisons is using different rules from other
> comparison operations
> --
>
> Key: HIVE-20204
> URL: https://issues.apache.org/jira/browse/HIVE-20204
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20204.1.patch
>
>
> Noticed this while looking at HIVE-20082.
> The type conversion done during GenericUDFIn (via
> ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(),
> whereas the other comparison operators (=, <, >, <=, >=) use
> FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN
> ('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1'
> would compare the values as doubles. This makes a difference for HIVE-20082
> since it is related to changing the 0-padding during decimal-to-string
> conversions.
> cc [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20198) Constant time table drops/renames
[ https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548628#comment-16548628 ] Eugene Koifman commented on HIVE-20198: --- could TBLS.TBL_ID be used as this ID? Not strictly related, but it would be nice if Table object contained this TBL_ID as well. > Constant time table drops/renames > - > > Key: HIVE-20198 > URL: https://issues.apache.org/jira/browse/HIVE-20198 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Priority: Major > > Currently table drops and table renames have O(P) performance (where P is the > number of partitions). When a managed table is deleted, the implementation > deletes table metadata and then deletes all partitions in HDFS. HDFS > operations are optimized and only do a sequential deletes for partitions > outside of table prefix. This operation is O(P)where Pis the number of > partitions. > Table rename goes through the list of partitions and modifies table name (and > potentially db name) in each partition. It also modifies each partition > location to match the new db/table name and renames directories (which is a > non-atomic and slow operation on S3). This is O(P) operation where P is the > number of partitions. > Basic idea is to do the following: > # Assign unique ID to each table > # Create directory name based on unique ID rather then the name > # Table rename then becomes metadata-only operation - there is no need to > change any location information. > # Table drop can become an asynchronous operation where the table is marked > as "deleted". Subsequent public metadata APIs should skip such tables. A > background cleaner thread may then go and clean up directories. > Since the table location is unique for each table, new tables will not reuse > existing locations. This change isn't compatible with the current behavior > where there is an assumption that table location is based on table name. We > can get around this by providing "opt-in" mechanism - special table property > that tells that the table can have such new behavior, so the improvement will > initially work for new tables created with this feature enabled. We may later > provide some tool to convert existing tables to the new scheme. > One complication is there in case where impersonation is enabled - the FS > operations should be performed using client UGI rather then server's, so the > cleaner thread should be able to use client UGIs. > Initially we can punt on this and do standard table drops when impersonation > is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20164) Murmur Hash : Make sure CTAS and IAS use correct bucketing version
[
https://issues.apache.org/jira/browse/HIVE-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548621#comment-16548621
]
Hive QA commented on HIVE-20164:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 39 unchanged - 0 fixed
= 43 total (was 39) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12682/dev-support/hive-personality.sh
|
| git revision | master / d5c589b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12682/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12682/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Murmur Hash : Make sure CTAS and IAS use correct bucketing version
> --
>
> Key: HIVE-20164
> URL: https://issues.apache.org/jira/browse/HIVE-20164
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-20164.1.patch, HIVE-20164.2.patch,
> HIVE-20164.3.patch
>
>
> With the migration to Murmur hash, CTAS and IAS from old table version to new
> table version does not work as intended and data is hashed using old hash
> logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548618#comment-16548618
]
Eugene Koifman edited comment on HIVE-18453 at 7/19/18 12:33 AM:
-
[~ikryvenko],
I don't think this is going to work. I don't think you are actually creating
transactional tables in your tests.
for example {{update transactional_table_test set value='foo';}} in
create_transactional.q would fail.
{noformat}
2018-07-18T17:25:35,181 ERROR [25cc35df-5e66-4d10-b31e-22a490cef829 main]
parse.UpdateDeleteSemanticAnalyzer: org.apache.hadoop.hive.ql.parse.SemanticE\
xception: Attempt to do update or delete on table
default.transactional_table_test that is not transactional
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2297)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2088)
{noformat}
Incidentally, in SemanticAnalyzer
{{tblProps = addDefaultProperties(tblProps, isExt, storageFormat,
dbDotTab, sortCols, isMaterialization, true);}}
why is the last param true?
I think you'd need to add another param to addDefaultProperties() to indicate
that it's called because of "create transactional" so that this method acts as
if CREATE_TABLES_AS_ACID and HIVE_CREATE_TABLES_AS_INSERT_ONLY are both true.
was (Author: ekoifman):
[~ikryvenko],
I don't think this is going to work. I don't think you are actually creating
transactional tables in your tests.
for example {{update transactional_table_test set value='foo';}} in
create_transactional.q would fail.
Incidentally, in SemanticAnalyzer
{{tblProps = addDefaultProperties(tblProps, isExt, storageFormat,
dbDotTab, sortCols, isMaterialization, true);}}
why is the last param true?
I think you'd need to add another param to addDefaultProperties() to indicate
that it's called because of "create transactional" so that this method acts as
if CREATE_TABLES_AS_ACID and HIVE_CREATE_TABLES_AS_INSERT_ONLY are both true.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548618#comment-16548618
]
Eugene Koifman commented on HIVE-18453:
---
[~ikryvenko],
I don't think this is going to work. I don't think you are actually creating
transactional tables in your tests.
for example {{update transactional_table_test set value='foo';}} in
create_transactional.q would fail.
Incidentally, in SemanticAnalyzer
{{tblProps = addDefaultProperties(tblProps, isExt, storageFormat,
dbDotTab, sortCols, isMaterialization, true);}}
why is the last param true?
I think you'd need to add another param to addDefaultProperties() to indicate
that it's called because of "create transactional" so that this method acts as
if CREATE_TABLES_AS_ACID and HIVE_CREATE_TABLES_AS_INSERT_ONLY are both true.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19924) Tag distcp jobs run by Repl Load
[
https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548593#comment-16548593
]
Hive QA commented on HIVE-19924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932029/HIVE-19924.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 14663 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacro
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroDoesNotExist
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroExistsDoNotIgnoreErrors
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExists
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistentWithIfExistsDoNotIgnoreNonExistent
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testOneInputParamters
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testThreeInputParamters
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testTwoInputParamters
(batchId=288)
org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testZeroInputParamters
(batchId=288)
org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryId (batchId=249)
org.apache.hive.jdbc.TestJdbcDriver2.testReplErrorScenarios (batchId=249)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testKillQuery (batchId=250)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=250)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testKillQuery (batchId=247)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=250)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=250)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=250)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerDAGTotalTasks
(batchId=247)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerSlowQueryExecutionTime
(batchId=247)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerTotalLaunchedTasks
(batchId=247)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.testTriggerVertexTotalTasks
(batchId=247)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers1
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomReadOps
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagRawInputSplitsKill
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagTotalTasks
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDefaultRawInputSplits
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesWrite
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerShortQueryElapsedTime
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerTotalTasks
(batchId=250)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsKill
(batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12681/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12681/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12681/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932029 -
[jira] [Comment Edited] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548561#comment-16548561
]
Jesus Camacho Rodriguez edited comment on HIVE-20095 at 7/18/18 11:49 PM:
--
[~msydoron], we need to fix those two test failures before committing this
patch. What is the error that we are getting?
After checking the code in more detail.
For the existing failures, I think your new code in JdbcRecordIterator is the
right way to go, but we should modify the {{deserialize}} method in JDBCSerde
to create the right type requested for Hive (the deserialize method in
DruidSerDe may serve as an example
https://github.com/apache/hive/blob/master/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java#L387
). JDBCSerde always has information for the columnTypes derived either from
from metadata query or from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, hence it should be straightforward. At the same
time, issue for nulls and timestamp would be fixed this way. What do you think?
was (Author: jcamachorodriguez):
[~msydoron], we need to fix those two test failures before committing this
patch. What is the error that we are getting?
For the existing failures, could we 1) modify {{getColumnTypes}} to obtain the
types from metadata query or from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), and 3) keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Would that work?
Alternatively, the fix for the long/double/timestamp can be fixed in a separate
ticket, then please create a follow-up for that.
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20207) Vectorization: Fix NULL / Wrong Results issues in Filter / Compare
[ https://issues.apache.org/jira/browse/HIVE-20207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-20207: --- > Vectorization: Fix NULL / Wrong Results issues in Filter / Compare > -- > > Key: HIVE-20207 > URL: https://issues.apache.org/jira/browse/HIVE-20207 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Write new UT tests that use random data and intentional isRepeating batches > to checks for NULL and Wrong Results for vectorized filter and compare. > BUGS: > 1) LongColLessLongColumn SIMD optimization do not work for very large > integers: > -7272907770454997143 < 8976171455044006767 > outputVector[i] = (vector1[i] - vector2[i]) >>> 63; > Produces 0 instead of 1... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548580#comment-16548580 ] Ashutosh Chauhan commented on HIVE-20205: - +1 pending tests .. yeah should go in branch-3 as well. > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548561#comment-16548561
]
Jesus Camacho Rodriguez edited comment on HIVE-20095 at 7/18/18 11:42 PM:
--
[~msydoron], we need to fix those two test failures before committing this
patch. What is the error that we are getting?
For the existing failures, could we 1) modify {{getColumnTypes}} to obtain the
types from metadata query or from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), and 3) keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Would that work?
Alternatively, the fix for the long/double/timestamp can be fixed in a separate
ticket, then please create a follow-up for that.
was (Author: jcamachorodriguez):
[~msydoron], we need to fix those two test failures before committing this
patch. For the existing failures, could we 1) modify {{getColumnTypes}} to
obtain the types from metadata query or from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), and 3) keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Alternatively, the
fix for the long/double/timestamp can be fixed in a separate ticket, then
please create a follow-up for that.
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[ https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548577#comment-16548577 ] Vihang Karajgaonkar commented on HIVE-20188: +1 LGTM > Split server-specific code outside of standalone metastore-common > - > > Key: HIVE-20188 > URL: https://issues.apache.org/jira/browse/HIVE-20188 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-20188.01.patch > > > The goal of this JIRA is to split metastore-common and separate the server > code into a separate module. This is still a pom-only change so all consumers > will have access to both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548575#comment-16548575 ] Sahil Takiar commented on HIVE-18831: - Yes, fix version is 4.0.0, thanks for updating. > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, > HIVE-18831.7.patch, HIVE-18831.8.WIP.patch, HIVE-18831.9.patch, > HIVE-18831.90.patch, HIVE-18831.91.patch, HIVE-18831.92.patch, > HIVE-18831.93.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20091) Tez: Add security credentials for FileSinkOperator output
[ https://issues.apache.org/jira/browse/HIVE-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548572#comment-16548572 ] Vineet Garg commented on HIVE-20091: I don't see this in branch-3.1 so removing fix version 3.1.0 and adding 3.2.0 > Tez: Add security credentials for FileSinkOperator output > - > > Key: HIVE-20091 > URL: https://issues.apache.org/jira/browse/HIVE-20091 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20091.01.patch, HIVE-20091.02.patch, > HIVE-20091.03.patch, HIVE-20091.04.patch, HIVE-20091.05.patch, > HIVE-20091.06.patch, HIVE-20091.07.patch, HIVE-20091.08.patch > > > DagUtils needs to add security credentials for the output for the > FileSinkOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20091) Tez: Add security credentials for FileSinkOperator output
[ https://issues.apache.org/jira/browse/HIVE-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20091: --- Fix Version/s: (was: 3.1.0) 3.2.0 > Tez: Add security credentials for FileSinkOperator output > - > > Key: HIVE-20091 > URL: https://issues.apache.org/jira/browse/HIVE-20091 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20091.01.patch, HIVE-20091.02.patch, > HIVE-20091.03.patch, HIVE-20091.04.patch, HIVE-20091.05.patch, > HIVE-20091.06.patch, HIVE-20091.07.patch, HIVE-20091.08.patch > > > DagUtils needs to add security credentials for the output for the > FileSinkOperator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19924) Tag distcp jobs run by Repl Load
[
https://issues.apache.org/jira/browse/HIVE-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548568#comment-16548568
]
Hive QA commented on HIVE-19924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 55 unchanged - 12
fixed = 56 total (was 67) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 2 new + 123 unchanged - 0
fixed = 125 total (was 123) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} service generated 1 new + 48 unchanged - 0 fixed = 49
total (was 48) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:service |
| | Redundant nullcheck of operation, which is known to be non-null in
org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationHandle)
Redundant null check at OperationManager.java:is known to be non-null in
org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationHandle)
Redundant null check at OperationManager.java:[line 306] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12681/dev-support/hive-personality.sh
|
| git revision | master / da1f758 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12681/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12681/yetus/diff-checkstyle-service.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12681/yetus/new-findbugs-service.html
|
| modules | C: ql service itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12681/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tag distcp jobs run by
[jira] [Updated] (HIVE-19357) Vectorization: assert_true HiveException erroneously gets suppressed to NULL
[
https://issues.apache.org/jira/browse/HIVE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19357:
Fix Version/s: 4.0.0
> Vectorization: assert_true HiveException erroneously gets suppressed to NULL
>
>
> Key: HIVE-19357
> URL: https://issues.apache.org/jira/browse/HIVE-19357
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zoltan Haindrich
>Assignee: Matt McCline
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19357.01.patch, HIVE-19357.02.patch,
> HIVE-19357.03.patch, HIVE-19357.04.patch, HIVE-19357.05.patch,
> HIVE-19357.06.patch
>
>
> This could be limited to assert exceptions; but might interfere with other
> exceptions...discovered while "fixing" testreopt after HIVE-19269
> {code}
> create table tu(id_uv int,id_uw int,u int);
> create table tv(id_uv int,v int);
> create table tw(id_uw int,w int);
> insert into tu values
> (10,10,10),(1,1,1),(2,2,2),(3,3,3),(4,4,4),(5,5,5),(6,6,6);
> insert into tv values (10,10),(1,1),(2,2),(3,3);
> insert into tw values
> (10,10),(1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9);
> set zzz=0;
> set hive.vectorized.execution.enabled=false;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- fails as expected
> set hive.vectorized.execution.enabled=true;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- there is a result set
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20115) Acid tables should not use footer scan for analyze
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548566#comment-16548566 ] Sergey Shelukhin commented on HIVE-20115: - RB > Acid tables should not use footer scan for analyze > -- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > > Discovered via incorrect stats in acid_no_buckets test on master-txnstats > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20115) Acid tables should not use footer scan for analyze
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548566#comment-16548566 ] Sergey Shelukhin edited comment on HIVE-20115 at 7/18/18 11:19 PM: --- RB https://reviews.apache.org/r/67969/ was (Author: sershe): RB > Acid tables should not use footer scan for analyze > -- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > > Discovered via incorrect stats in acid_no_buckets test on master-txnstats > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548561#comment-16548561
]
Jesus Camacho Rodriguez edited comment on HIVE-20095 at 7/18/18 11:18 PM:
--
[~msydoron], we need to fix those two test failures before committing this
patch. For the existing failures, could we 1) modify {{getColumnTypes}} to
obtain the types from metadata query or from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), and 3) keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Alternatively, the
fix for the long/double/timestamp can be fixed in a separate ticket, then
please create a follow-up for that.
was (Author: jcamachorodriguez):
[~msydoron], we need to fix those two test failures before committing this
patch. For the existing failures, could we 1) modify {{getColumnTypes}} to
obtain the types with metadata query vs from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), 3) and keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Alternatively, the
fix for the long/double/timestamp can be fixed in a separate ticket, then
please create a follow-up for that.
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20115) Acid tables should not use footer scan for analyze
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20115: Status: Patch Available (was: Open) [~ekoifman] can you take a look? This is a master patch > Acid tables should not use footer scan for analyze > -- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > > Discovered via incorrect stats in acid_no_buckets test on master-txnstats > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20095) Fix jdbc external table feature
[
https://issues.apache.org/jira/browse/HIVE-20095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548561#comment-16548561
]
Jesus Camacho Rodriguez commented on HIVE-20095:
[~msydoron], we need to fix those two test failures before committing this
patch. For the existing failures, could we 1) modify {{getColumnTypes}} to
obtain the types with metadata query vs from conf depending on the existence of
{{Constants.HIVE_JDBC_QUERY}}, 2) then use the method to pass the types to the
constructor of JdbcRecordIterator (as it is currently done), 3) and keep the
code to infer from those types the object that we need to extract for each row,
possibly fixing the issue with null values and timestamp? Alternatively, the
fix for the long/double/timestamp can be fixed in a separate ticket, then
please create a follow-up for that.
> Fix jdbc external table feature
> ---
>
> Key: HIVE-20095
> URL: https://issues.apache.org/jira/browse/HIVE-20095
> Project: Hive
> Issue Type: Bug
>Reporter: Jonathan Doron
>Assignee: Jonathan Doron
>Priority: Major
> Attachments: HIVE-20095.1.patch, HIVE-20095.2.patch,
> HIVE-20095.3.patch, HIVE-20095.4.patch
>
>
> It seems like the committed code for HIVE-19161
> (7584b3276bebf64aa006eaa162c0a6264d8fcb56) reverted some of HIVE-18423
> updates, and therefore some of the external table queries are not working
> correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20115: Attachment: HIVE-20115.patch > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20115) Acid tables should not use footer scan for analyze
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20115: Summary: Acid tables should not use footer scan for analyze (was: acid_no_buckets.q fails) > Acid tables should not use footer scan for analyze > -- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20115) Acid tables should not use footer scan for analyze
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20115: Description: Discovered via incorrect stats in acid_no_buckets test on master-txnstats branch > Acid tables should not use footer scan for analyze > -- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20115.patch > > > Discovered via incorrect stats in acid_no_buckets test on master-txnstats > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-20205: - Status: Patch Available (was: Open) [~ashutoshc] Could you please review? Is there room to push this to branch-3 as well ? > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20164) Murmur Hash : Make sure CTAS and IAS use correct bucketing version
[ https://issues.apache.org/jira/browse/HIVE-20164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20164: -- Attachment: HIVE-20164.3.patch > Murmur Hash : Make sure CTAS and IAS use correct bucketing version > -- > > Key: HIVE-20164 > URL: https://issues.apache.org/jira/browse/HIVE-20164 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20164.1.patch, HIVE-20164.2.patch, > HIVE-20164.3.patch > > > With the migration to Murmur hash, CTAS and IAS from old table version to new > table version does not work as intended and data is hashed using old hash > logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-20205: - Attachment: HIVE-20205.patch > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548543#comment-16548543 ] Sergey Shelukhin commented on HIVE-20115: - BasicStatsNoJob task cannot be used for ACID, that's the problem. > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548533#comment-16548533 ] Vineet Garg commented on HIVE-18831: [~stakiar] Can you confirm the fix version? I don't see this in branch-3 so I assume this went it to master only for 4.0. > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, > HIVE-18831.7.patch, HIVE-18831.8.WIP.patch, HIVE-18831.9.patch, > HIVE-18831.90.patch, HIVE-18831.91.patch, HIVE-18831.92.patch, > HIVE-18831.93.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18831: --- Fix Version/s: 4.0.0 > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, > HIVE-18831.7.patch, HIVE-18831.8.WIP.patch, HIVE-18831.9.patch, > HIVE-18831.90.patch, HIVE-18831.91.patch, HIVE-18831.92.patch, > HIVE-18831.93.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[ https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548524#comment-16548524 ] Vineet Garg commented on HIVE-19252: [~vgumashta] Can you confirm and add the fix version? > TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently > > > Key: HIVE-19252 > URL: https://issues.apache.org/jira/browse/HIVE-19252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch > > > For last 8 builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[ https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19252: --- Fix Version/s: 3.1.0 > TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently > > > Key: HIVE-19252 > URL: https://issues.apache.org/jira/browse/HIVE-19252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch > > > For last 8 builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20199) Improved filtering performance for a large number of partitions in a single table.
[
https://issues.apache.org/jira/browse/HIVE-20199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548523#comment-16548523
]
Hive QA commented on HIVE-20199:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932064/021-HIVE-20199.mysql.sql
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12680/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12680/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12680/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-18 22:49:05.294
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12680/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-18 22:49:05.297
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at da1f758 HIVE-20172 : StatsUpdater failed with GSS Exception
while trying to connect to remote metastore (Rajkumar Singh via Ashutosh
Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at da1f758 HIVE-20172 : StatsUpdater failed with GSS Exception
while trying to connect to remote metastore (Rajkumar Singh via Ashutosh
Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-18 22:49:06.735
+ rm -rf ../yetus_PreCommit-HIVE-Build-12680
+ mkdir ../yetus_PreCommit-HIVE-Build-12680
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12680
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12680/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12680
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932064 - PreCommit-HIVE-Build
> Improved filtering performance for a large number of partitions in a single
> table.
> --
>
> Key: HIVE-20199
> URL: https://issues.apache.org/jira/browse/HIVE-20199
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Biao Wu
>Assignee: Biao Wu
>Priority: Major
> Attachments: 021-HIVE-20199.mysql.sql
>
>
> eg:
> {code:sql}
> select * from test where dt = '20180606'
> {code}
> The filter 'dt=20180606' will be pushed down to mysql for execution, but the
> test table contains a large number of partitions, this is a poor performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19385) Optional hive env variable to redirect bin/hive to use Beeline
[ https://issues.apache.org/jira/browse/HIVE-19385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19385: --- Fix Version/s: 3.1.0 > Optional hive env variable to redirect bin/hive to use Beeline > -- > > Key: HIVE-19385 > URL: https://issues.apache.org/jira/browse/HIVE-19385 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19385.1.patch > > > With beeline-site and beeline-user-site, the user can easily specify default > hs2 urls to connect. We can use an optional env variable, which when set, > will enable bin/hive to use beeline. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19385) Optional hive env variable to redirect bin/hive to use Beeline
[ https://issues.apache.org/jira/browse/HIVE-19385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548517#comment-16548517 ] Vineet Garg commented on HIVE-19385: [~vgumashta] [~daijy] Can you confirm and add the fix version? > Optional hive env variable to redirect bin/hive to use Beeline > -- > > Key: HIVE-19385 > URL: https://issues.apache.org/jira/browse/HIVE-19385 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19385.1.patch > > > With beeline-site and beeline-user-site, the user can easily specify default > hs2 urls to connect. We can use an optional env variable, which when set, > will enable bin/hive to use beeline. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548516#comment-16548516
]
Hive QA commented on HIVE-20192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932101/HIVE-20192.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12679/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12679/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12679/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932101/HIVE-20192.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932101 - PreCommit-HIVE-Build
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548514#comment-16548514
]
Hive QA commented on HIVE-18453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932065/HIVE-18453.05.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14666 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
(batchId=297)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12678/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12678/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12678/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932065 - PreCommit-HIVE-Build
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19357) Vectorization: assert_true HiveException erroneously gets suppressed to NULL
[
https://issues.apache.org/jira/browse/HIVE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548515#comment-16548515
]
Vineet Garg commented on HIVE-19357:
[~mmccline] Can you add the fix version? I have added the 3.1.0 since it went
to branch-3.1.0 but I am not sure what other branches this went to.
> Vectorization: assert_true HiveException erroneously gets suppressed to NULL
>
>
> Key: HIVE-19357
> URL: https://issues.apache.org/jira/browse/HIVE-19357
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zoltan Haindrich
>Assignee: Matt McCline
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19357.01.patch, HIVE-19357.02.patch,
> HIVE-19357.03.patch, HIVE-19357.04.patch, HIVE-19357.05.patch,
> HIVE-19357.06.patch
>
>
> This could be limited to assert exceptions; but might interfere with other
> exceptions...discovered while "fixing" testreopt after HIVE-19269
> {code}
> create table tu(id_uv int,id_uw int,u int);
> create table tv(id_uv int,v int);
> create table tw(id_uw int,w int);
> insert into tu values
> (10,10,10),(1,1,1),(2,2,2),(3,3,3),(4,4,4),(5,5,5),(6,6,6);
> insert into tv values (10,10),(1,1),(2,2),(3,3);
> insert into tw values
> (10,10),(1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9);
> set zzz=0;
> set hive.vectorized.execution.enabled=false;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- fails as expected
> set hive.vectorized.execution.enabled=true;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- there is a result set
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19389) Schematool: For Hive's Information Schema, use embedded HS2 as default
[ https://issues.apache.org/jira/browse/HIVE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19389: --- Fix Version/s: 3.1.0 > Schematool: For Hive's Information Schema, use embedded HS2 as default > -- > > Key: HIVE-19389 > URL: https://issues.apache.org/jira/browse/HIVE-19389 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19389.1.patch, HIVE-19389.2.patch, > HIVE-19389.2.patch > > > Currently, for initializing/upgrading Hive's information schema, we require a > full jdbc url (for HS2). It will be good to have it connect using embedded > HS2 by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19357) Vectorization: assert_true HiveException erroneously gets suppressed to NULL
[
https://issues.apache.org/jira/browse/HIVE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19357:
---
Fix Version/s: 3.1.0
> Vectorization: assert_true HiveException erroneously gets suppressed to NULL
>
>
> Key: HIVE-19357
> URL: https://issues.apache.org/jira/browse/HIVE-19357
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zoltan Haindrich
>Assignee: Matt McCline
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19357.01.patch, HIVE-19357.02.patch,
> HIVE-19357.03.patch, HIVE-19357.04.patch, HIVE-19357.05.patch,
> HIVE-19357.06.patch
>
>
> This could be limited to assert exceptions; but might interfere with other
> exceptions...discovered while "fixing" testreopt after HIVE-19269
> {code}
> create table tu(id_uv int,id_uw int,u int);
> create table tv(id_uv int,v int);
> create table tw(id_uw int,w int);
> insert into tu values
> (10,10,10),(1,1,1),(2,2,2),(3,3,3),(4,4,4),(5,5,5),(6,6,6);
> insert into tv values (10,10),(1,1),(2,2),(3,3);
> insert into tw values
> (10,10),(1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9);
> set zzz=0;
> set hive.vectorized.execution.enabled=false;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- fails as expected
> set hive.vectorized.execution.enabled=true;
> select assert_true(${hiveconf:zzz}>sum(1)) from tu join tv on
> (tu.id_uv=tv.id_uv) where u<10 and v>1;
> -- there is a result set
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548480#comment-16548480 ] Sergey Shelukhin edited comment on HIVE-20115 at 7/18/18 10:22 PM: --- The problem is not related to the changes on the branch but merely exposed by it. Stats on the table become wrong after an update followed by "analyze table srcpart_acid PARTITION(ds, hr) compute statistics;" (stats after update are valid) So, ACID analyze appears to be broken, but this wasn't exposed previously since we weren't using stats for queries. cc [~ekoifman] If analyze is removed, the result is correct. I'm looking at why analyze is such as it is. Probably some stupid clever metastore side update, or something. was (Author: sershe): The problem is not related to the changes on the branch but merely exposed by it. Stats on the table become wrong after an update followed by "analyze table srcpart_acid PARTITION(ds, hr) compute statistics;" (stats after update are valid) After that the wrong stats are adjusted by various queries (garbage in, garbage out). If analyze is removed, the result is correct. I'm looking at why analyze is such as it is. Probably some stupid clever metastore side update, or something. > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548480#comment-16548480 ] Sergey Shelukhin edited comment on HIVE-20115 at 7/18/18 10:19 PM: --- The problem is not related to the changes on the branch but merely exposed by it. Stats on the table become wrong after an update followed by "analyze table srcpart_acid PARTITION(ds, hr) compute statistics;" (stats after update are valid) After that the wrong stats are adjusted by various queries (garbage in, garbage out). If analyze is removed, the result is correct. I'm looking at why analyze is such as it is. Probably some stupid clever metastore side update, or something. was (Author: sershe): The problem is not related to the change but merely exposed by it. Stats on the table become wrong after an update followed by "analyze table srcpart_acid PARTITION(ds, hr) compute statistics;" After that the wrong stats are adjusted by various queries (garbage in, garbage out). If analyze is removed, the result is correct. I'm looking at why analyze is such as it is. Probably some stupid clever metastore side update, or something. > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548481#comment-16548481 ] Jesus Camacho Rodriguez commented on HIVE-17896: [~teddy.choi], thanks for applying the changes. Could you regenerate those q files and update RB to check them too? Latest patch LGTM, unless [~mmccline] or [~gopalv] have any comment about the vectorized operator. > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.12.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch, > HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548480#comment-16548480 ] Sergey Shelukhin commented on HIVE-20115: - The problem is not related to the change but merely exposed by it. Stats on the table become wrong after an update followed by "analyze table srcpart_acid PARTITION(ds, hr) compute statistics;" After that the wrong stats are adjusted by various queries (garbage in, garbage out). If analyze is removed, the result is correct. I'm looking at why analyze is such as it is. Probably some stupid clever metastore side update, or something. > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548465#comment-16548465
]
Hive QA commented on HIVE-18453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
10s{color} | {color:red} ql generated 4 new + 2270 unchanged - 4 fixed = 2274
total (was 2274) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA227 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46194-46207] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA271 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46544-46557] |
| | Dead store to LA28_141 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At HiveParser.java:[line 46024] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA286 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46680-46693] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12678/dev-support/hive-personality.sh
|
| git revision | master / da1f758 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12678/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12678/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch, HIVE-18453.05.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to
[jira] [Assigned] (HIVE-20115) acid_no_buckets.q fails
[ https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20115: --- Assignee: Sergey Shelukhin (was: Steve Yeom) > acid_no_buckets.q fails > --- > > Key: HIVE-20115 > URL: https://issues.apache.org/jira/browse/HIVE-20115 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-19933:
-
Attachment: (was: HIVE-19933.1.patch)
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19933.1.patch
>
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alice Fan updated HIVE-19933:
-
Fix Version/s: 4.0.0
Attachment: HIVE-19933.1.patch
Status: Patch Available (was: In Progress)
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19933.1.patch
>
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20061) add a config flag to turn off txn stats
[ https://issues.apache.org/jira/browse/HIVE-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548437#comment-16548437 ] Sergey Shelukhin edited comment on HIVE-20061 at 7/18/18 9:40 PM: -- Committed to branch; the setting implemented metastore side, and a test. was (Author: sershe): Committed to branch. > add a config flag to turn off txn stats > > > Key: HIVE-20061 > URL: https://issues.apache.org/jira/browse/HIVE-20061 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: txnstats > > Attachments: HIVE-20061.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20061) add a config flag to turn off txn stats
[ https://issues.apache.org/jira/browse/HIVE-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20061: Attachment: HIVE-20061.patch > add a config flag to turn off txn stats > > > Key: HIVE-20061 > URL: https://issues.apache.org/jira/browse/HIVE-20061 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: txnstats > > Attachments: HIVE-20061.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20193) cboInfo is not present in the explain plan json
[
https://issues.apache.org/jira/browse/HIVE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548438#comment-16548438
]
Hive QA commented on HIVE-20193:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932077/HIVE-20193.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 14662 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4] (batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join0] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_join0]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[plan_json] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6]
(batchId=44)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12677/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12677/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12677/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932077 - PreCommit-HIVE-Build
> cboInfo is not present in the explain plan json
> ---
>
> Key: HIVE-20193
> URL: https://issues.apache.org/jira/browse/HIVE-20193
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20193.1.patch
>
>
> *cboInfo* attribute is not present in the explain plan json that is provided
> to the pre exec hook in hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-20061) add a config flag to turn off txn stats
[ https://issues.apache.org/jira/browse/HIVE-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-20061. - Resolution: Fixed Fix Version/s: txnstats Committed to branch. > add a config flag to turn off txn stats > > > Key: HIVE-20061 > URL: https://issues.apache.org/jira/browse/HIVE-20061 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: txnstats > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20193) cboInfo is not present in the explain plan json
[
https://issues.apache.org/jira/browse/HIVE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548401#comment-16548401
]
Hive QA commented on HIVE-20193:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 341 unchanged - 0
fixed = 342 total (was 341) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12677/dev-support/hive-personality.sh
|
| git revision | master / bac1d98 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12677/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12677/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> cboInfo is not present in the explain plan json
> ---
>
> Key: HIVE-20193
> URL: https://issues.apache.org/jira/browse/HIVE-20193
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20193.1.patch
>
>
> *cboInfo* attribute is not present in the explain plan json that is provided
> to the pre exec hook in hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20172) StatsUpdater failed with GSS Exception while trying to connect to remote metastore
[
https://issues.apache.org/jira/browse/HIVE-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20172:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Rajkumar!
> StatsUpdater failed with GSS Exception while trying to connect to remote
> metastore
> --
>
> Key: HIVE-20172
> URL: https://issues.apache.org/jira/browse/HIVE-20172
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Hive-1.2.1,Hive2.1,java8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20172.patch
>
>
> StatsUpdater task failed with GSS Exception while trying to connect to remote
> Metastore.
> {code}
> org.apache.thrift.transport.TTransportException: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
>
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:300)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> {code}
> since metastore client is running in HMS so there is no need to connect to
> remote URI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19933 started by Alice Fan.
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
> Attachments: HIVE-19933.1.patch
>
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17683) Annotate Query Plan with locking information
[
https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548382#comment-16548382
]
Eugene Koifman commented on HIVE-17683:
---
[~ikryvenko],
this looks good in general. One question:
Did you mean to add support for 'formatted' option? In patch 4, adding
'formatted' produces
{noformat}
{"LOCK INFORMATION:":"[]"}
{noformat}
{{ExplainTask.getLocks()}} has a bug
I think, {[explain locks drop table test_explain_locks}} produces
Read/WriteEntity because this table doesn't exist at the time this command runs.
> Annotate Query Plan with locking information
>
>
> Key: HIVE-17683
> URL: https://issues.apache.org/jira/browse/HIVE-17683
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Critical
> Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch,
> HIVE-17683.03.patch, HIVE-17683.04.patch
>
>
> Explore if it's possible to add info about what locks will be asked for to
> the query plan.
> Lock acquisition (for Acid Lock Manager) is done in
> DbTxnManager.acquireLocks() which is called once the query starts running.
> Would need to refactor that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548380#comment-16548380
]
Adesh Kumar Rao commented on HIVE-15131:
Uploaded new patch to fix the parquet_analyze test failure. This was happening
because in case of noscan stats collection, parquet reader gets a dummySplit
with start/length set to 0/0 in which case, using the filter api was returning
0 blocks for the dummySplit and hence wrong stats were updated.
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch, HIVE-15131.4.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Attachment: HIVE-15131.4.patch
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch, HIVE-15131.4.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Status: Open (was: Patch Available)
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch, HIVE-15131.4.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Status: Patch Available (was: Open)
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch, HIVE-15131.4.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Status: Open (was: Patch Available)
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Status: Patch Available (was: Open)
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15131) Change Parquet reader to read metadata on the task side
[
https://issues.apache.org/jira/browse/HIVE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adesh Kumar Rao updated HIVE-15131:
---
Attachment: HIVE-15131.3.patch
> Change Parquet reader to read metadata on the task side
> ---
>
> Key: HIVE-15131
> URL: https://issues.apache.org/jira/browse/HIVE-15131
> Project: Hive
> Issue Type: Bug
> Components: Reader
>Reporter: Chao Sun
>Assignee: Adesh Kumar Rao
>Priority: Major
> Attachments: HIVE-15131.1.patch, HIVE-15131.2.patch,
> HIVE-15131.3.patch
>
>
> Currently the {{ParquetRecordReaderWrapper}} still uses the {{readFooter}}
> API without filtering, which means it needs to read metadata about all row
> groups every time. This could some issues when input dataset is particularly
> big and has many columns.
> [Parquet-84|https://issues.apache.org/jira/browse/PARQUET-84] introduced
> another API which allows to do row group filtering on the task side. Hive
> should adopt this API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20188) Split server-specific code outside of standalone metastore-common
[
https://issues.apache.org/jira/browse/HIVE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548369#comment-16548369
]
Hive QA commented on HIVE-20188:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12931998/HIVE-20188.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12676/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12676/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12676/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12931998/HIVE-20188.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12931998 - PreCommit-HIVE-Build
> Split server-specific code outside of standalone metastore-common
> -
>
> Key: HIVE-20188
> URL: https://issues.apache.org/jira/browse/HIVE-20188
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20188.01.patch
>
>
> The goal of this JIRA is to split metastore-common and separate the server
> code into a separate module. This is still a pom-only change so all consumers
> will have access to both.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20194) HiveMetastoreClient should use reflection to instantiate embedded HMS instance
[
https://issues.apache.org/jira/browse/HIVE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548368#comment-16548368
]
Hive QA commented on HIVE-20194:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932012/HIVE-20194.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12675/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12675/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12675/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932012/HIVE-20194.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932012 - PreCommit-HIVE-Build
> HiveMetastoreClient should use reflection to instantiate embedded HMS instance
> --
>
> Key: HIVE-20194
> URL: https://issues.apache.org/jira/browse/HIVE-20194
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20194.01.patch
>
>
> When HiveMetastoreClient is used in embedded mode, it instantiates metastore
> server. Since we want to separate client and server code we can no longer
> instantiate the class directly but need to use reflection for that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20120) Hive incremental load dag genetration causing OOM error
[
https://issues.apache.org/jira/browse/HIVE-20120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548366#comment-16548366
]
Hive QA commented on HIVE-20120:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932067/HIVE-20120.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12673/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12673/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932067 - PreCommit-HIVE-Build
> Hive incremental load dag genetration causing OOM error
> ---
>
> Key: HIVE-20120
> URL: https://issues.apache.org/jira/browse/HIVE-20120
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20120.01.patch
>
>
> Split the incremental load into multiple iterations. In each iteration create
> number of tasks equal to the configured value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[ https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20202: - Summary: Add profiler endpoint to HS2 and LLAP (was: Add profiler endpoint to httpserver) > Add profiler endpoint to HS2 and LLAP > - > > Key: HIVE-20202 > URL: https://issues.apache.org/jira/browse/HIVE-20202 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch > > > Add a web endpoint for profiling based on async-profiler. This servlet should > be added to httpserver so that HS2 and LLAP daemons can output flamegraphs > when their /prof endpoint is hit. Since this will be based on > [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock > contentions, HW counters etc. will also be supported in addition to cpu > profiling. In most cases the profiling overhead is pretty low and is safe to > run on production. More analysis on CPU and memory overhead here > [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and > [https://github.com/jvm-profiling-tools/async-profiler/issues/131] > > For the impatient, here is the usage doc and the sample output > [https://github.com/prasanthj/nightswatch/blob/master/README.md] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20172) StatsUpdater failed with GSS Exception while trying to connect to remote metastore
[
https://issues.apache.org/jira/browse/HIVE-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548345#comment-16548345
]
Sergey Shelukhin commented on HIVE-20172:
-
StatsUpdater on master is a separate component (in addition to compactor
updating stats), so the description is confusing.
> StatsUpdater failed with GSS Exception while trying to connect to remote
> metastore
> --
>
> Key: HIVE-20172
> URL: https://issues.apache.org/jira/browse/HIVE-20172
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Hive-1.2.1,Hive2.1,java8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20172.patch
>
>
> StatsUpdater task failed with GSS Exception while trying to connect to remote
> Metastore.
> {code}
> org.apache.thrift.transport.TTransportException: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
>
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:300)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> {code}
> since metastore client is running in HMS so there is no need to connect to
> remote URI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20172) StatsUpdater failed with GSS Exception while trying to connect to remote metastore
[
https://issues.apache.org/jira/browse/HIVE-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20172:
--
Summary: StatsUpdater failed with GSS Exception while trying to connect to
remote metastore (was: compactor failed with GSS Exception while trying to
connect to remote metastore)
> StatsUpdater failed with GSS Exception while trying to connect to remote
> metastore
> --
>
> Key: HIVE-20172
> URL: https://issues.apache.org/jira/browse/HIVE-20172
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Hive-1.2.1,Hive2.1,java8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20172.patch
>
>
> StatsUpdater task failed with GSS Exception while trying to connect to remote
> Metastore.
> {code}
> org.apache.thrift.transport.TTransportException: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
>
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:300)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> {code}
> since metastore client is running in HMS so there is no need to connect to
> remote URI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20172) compactor failed with GSS Exception while trying to connect to remote metastore
[
https://issues.apache.org/jira/browse/HIVE-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548339#comment-16548339
]
Rajkumar Singh commented on HIVE-20172:
---
once the compaction job is done, statsupdate run the gatherStats which is
failing, the change is inside gatherStats only, so compaction job is not
failing but stats update failed in kerberos environment, fixing the jira title.
> compactor failed with GSS Exception while trying to connect to remote
> metastore
> ---
>
> Key: HIVE-20172
> URL: https://issues.apache.org/jira/browse/HIVE-20172
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Hive-1.2.1,Hive2.1,java8
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20172.patch
>
>
> StatsUpdater task failed with GSS Exception while trying to connect to remote
> Metastore.
> {code}
> org.apache.thrift.transport.TTransportException: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
>
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:92)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
>
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3526)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3558)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:533)
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker$StatsUpdater.gatherStats(Worker.java:300)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:265)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
> )
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:534)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:282)
>
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:76)
>
> {code}
> since metastore client is running in HMS so there is no need to connect to
> remote URI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20120) Hive incremental load dag genetration causing OOM error
[
https://issues.apache.org/jira/browse/HIVE-20120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548338#comment-16548338
]
Hive QA commented on HIVE-20120:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2274 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 1 new + 2273 unchanged - 1 fixed = 2274
total (was 2274) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork) At IncrementalLoadTasksBuilder.java:from instance
method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork) At IncrementalLoadTasksBuilder.java:[line 94] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12673/dev-support/hive-personality.sh
|
| git revision | master / bac1d98 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12673/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12673/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive incremental load dag genetration causing OOM error
> ---
>
> Key: HIVE-20120
> URL: https://issues.apache.org/jira/browse/HIVE-20120
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20120.01.patch
>
>
> Split the incremental load into multiple iterations. In each iteration create
> number of tasks equal to the configured value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
