[jira] [Commented] (HIVE-21426) Remove Utilities Global Random
[
https://issues.apache.org/jira/browse/HIVE-21426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790146#comment-16790146
]
Hive QA commented on HIVE-21426:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 139 unchanged - 1
fixed = 139 total (was 140) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} ql generated 0 new + 2256 unchanged - 2 fixed = 2256
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16456/dev-support/hive-personality.sh
|
| git revision | master / 9f2f101 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16456/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Utilities Global Random
> --
>
> Key: HIVE-21426
> URL: https://issues.apache.org/jira/browse/HIVE-21426
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21426.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L253
> Remove global {{Random}} object in favor of {{ThreadLocalRandom}}.
> {quote}
> ThreadLocalRandom is initialized with an internally generated seed that may
> not otherwise be modified. When applicable, use of ThreadLocalRandom rather
> than shared Random objects in concurrent programs will typically encounter
> much less overhead and contention.
> {quote}
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadLocalRandom.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21426) Remove Utilities Global Random
[
https://issues.apache.org/jira/browse/HIVE-21426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790176#comment-16790176
]
Hive QA commented on HIVE-21426:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962021/HIVE-21426.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16456/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16456/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16456/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962021 - PreCommit-HIVE-Build
> Remove Utilities Global Random
> --
>
> Key: HIVE-21426
> URL: https://issues.apache.org/jira/browse/HIVE-21426
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21426.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L253
> Remove global {{Random}} object in favor of {{ThreadLocalRandom}}.
> {quote}
> ThreadLocalRandom is initialized with an internally generated seed that may
> not otherwise be modified. When applicable, use of ThreadLocalRandom rather
> than shared Random objects in concurrent programs will typically encounter
> much less overhead and contention.
> {quote}
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadLocalRandom.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21425) Use DirectExecutorService for getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790239#comment-16790239
]
Hive QA commented on HIVE-21425:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 112 unchanged - 1
fixed = 112 total (was 113) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16459/dev-support/hive-personality.sh
|
| git revision | master / 9f2f101 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16459/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use DirectExecutorService for getInputSummary
> -
>
> Key: HIVE-21425
> URL: https://issues.apache.org/jira/browse/HIVE-21425
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21425.1.patch, HIVE-21425.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|title=Utilities.java}
> int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
> pathNeedProcess.size());
> if (numExecutors > 1) {
> LOG.info("Using {} threads for getContentSummary", numExecutors);
> executor = Executors.newFixedThreadPool(numExecutors,
> new ThreadFactoryBuilder().setDaemon(true)
> .setNameFormat("Get-Input-Summary-%d").build());
> } else {
> executor = null;
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
> Instead of using a 'null' {{ExecutorService}}, use Guava's
> {{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21425) Use DirectExecutorService for getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790253#comment-16790253
]
Hive QA commented on HIVE-21425:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962051/HIVE-21425.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
org.apache.hive.jdbc.TestSSL.testSSLFetch (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16459/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16459/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962051 - PreCommit-HIVE-Build
> Use DirectExecutorService for getInputSummary
> -
>
> Key: HIVE-21425
> URL: https://issues.apache.org/jira/browse/HIVE-21425
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21425.1.patch, HIVE-21425.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|title=Utilities.java}
> int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
> pathNeedProcess.size());
> if (numExecutors > 1) {
> LOG.info("Using {} threads for getContentSummary", numExecutors);
> executor = Executors.newFixedThreadPool(numExecutors,
> new ThreadFactoryBuilder().setDaemon(true)
> .setNameFormat("Get-Input-Summary-%d").build());
> } else {
> executor = null;
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
> Instead of using a 'null' {{ExecutorService}}, use Guava's
> {{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Attachment: HIVE-21382.3.patch
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch, HIVE-21382.3.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790224#comment-16790224
]
Hive QA commented on HIVE-21382:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962042/HIVE-21382.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16458/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-12 04:32:12.577
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16458/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-12 04:32:12.580
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9f2f101 HIVE-21388: Constant UDF is not pushed to JDBCStorage
Handler (Jesus Camacho Rodriguez, reviewed by Jason Dere)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9f2f101 HIVE-21388: Constant UDF is not pushed to JDBCStorage
Handler (Jesus Camacho Rodriguez, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-12 04:32:13.469
+ rm -rf ../yetus_PreCommit-HIVE-Build-16458
+ mkdir ../yetus_PreCommit-HIVE-Build-16458
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16458
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16458/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java:
does not exist in index
error: a/ql/src/test/queries/clientpositive/constraints_optimization.q: does
not exist in index
error:
a/ql/src/test/results/clientpositive/llap/constraints_optimization.q.out: does
not exist in index
error:
a/ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query23.q.out:
does not exist in index
error:
a/ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query39.q.out:
does not exist in index
error:
a/ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query64.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/perf/tez/constraints/query23.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/perf/tez/constraints/query39.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/perf/tez/constraints/query64.q.out:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc5885607783622954764.exe, --version]
protoc-jar: executing: [/tmp/protoc5885607783622954764.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
[jira] [Commented] (HIVE-21411) LEFT JOIN CONVERT TO INNER JOIN LEAD TO WRONG RESULT
[
https://issues.apache.org/jira/browse/HIVE-21411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790246#comment-16790246
]
Vineet Garg commented on HIVE-21411:
[~tianmenglan2016] Can you provide reproducer (including schema/data)?
> LEFT JOIN CONVERT TO INNER JOIN LEAD TO WRONG RESULT
>
>
> Key: HIVE-21411
> URL: https://issues.apache.org/jira/browse/HIVE-21411
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0, 2.3.0
>Reporter: xialu
>Assignee: Ashutosh Chauhan
>Priority: Critical
>
> when i have not assign alias table name to the left side table , left join
> convert to inner join. left side table's alias in ast tree called left.
> for example
> select nvl(ss_wholesale_cost, 10), d_quarter_name from lulu.store_sales left
> join lulu.date_dim on ss_sold_date_sk = d_date_sk limit 10;
> {noformat}
> | STAGE DEPENDENCIES: |
> | Stage-1 is a root stage |
> | Stage-0 depends on stages: Stage-1 |
> | |
> | STAGE PLANS: |
> | Stage: Stage-1 |
> | Map Reduce |
> | Map Operator Tree: |
> | TableScan |
> | alias: left |
> | Statistics: Num rows: 200 Data size: 28499 Basic stats: COMPLETE Column
> stats: NONE |
> | Filter Operator |
> | predicate: ss_sold_date_sk is not null (type: boolean) |
> | Statistics: Num rows: 200 Data size: 28499 Basic stats: COMPLETE Column
> stats: NONE |
> | Select Operator |
> | expressions: ss_wholesale_cost (type: decimal(7,2)), ss_sold_date_sk (type:
> bigint) |
> | outputColumnNames: _col0, _col1 |
> | Statistics: Num rows: 200 Data size: 28499 Basic stats: COMPLETE Column
> stats: NONE |
> | Reduce Output Operator |
> | key expressions: _col1 (type: bigint) |
> | sort order: + |
> | Map-reduce partition columns: _col1 (type: bigint) |
> | Statistics: Num rows: 200 Data size: 28499 Basic stats: COMPLETE Column
> stats: NONE |
> | value expressions: _col0 (type: decimal(7,2)) |
> | TableScan |
> | alias: date_dim |
> | Statistics: Num rows: 200 Data size: 25639 Basic stats: COMPLETE Column
> stats: NONE |
> | Filter Operator |
> | predicate: d_date_sk is not null (type: boolean) |
> | Statistics: Num rows: 200 Data size: 25639 Basic stats: COMPLETE Column
> stats: NONE |
> | Select Operator |
> | expressions: d_date_sk (type: bigint), d_quarter_name (type: string) |
> | outputColumnNames: _col0, _col1 |
> | Statistics: Num rows: 200 Data size: 25639 Basic stats: COMPLETE Column
> stats: NONE |
> | Reduce Output Operator |
> | key expressions: _col0 (type: bigint) |
> | sort order: + |
> | Map-reduce partition columns: _col0 (type: bigint) |
> | Statistics: Num rows: 200 Data size: 25639 Basic stats: COMPLETE Column
> stats: NONE |
> | value expressions: _col1 (type: string) |
> | Reduce Operator Tree: |
> | Join Operator |
> | condition map: |
> | Inner Join 0 to 1 |
> | keys: |
> | 0 _col1 (type: bigint) |
> | 1 _col0 (type: bigint) |
> | outputColumnNames: _col0, _col3 |
> | Statistics: Num rows: 220 Data size: 31348 Basic stats: COMPLETE Column
> stats: NONE |
> | Select Operator |
> | expressions: NVL(_col0,10) (type: decimal(12,2)), _col3 (type: string) |
> | outputColumnNames: _col0, _col1 |
> | Statistics: Num rows: 220 Data size: 31348 Basic stats: COMPLETE Column
> stats: NONE |
> | Limit |
> | Number of rows: 100 |
> | Statistics: Num rows: 100 Data size: 14200 Basic stats: COMPLETE Column
> stats: NONE |
> | File Output Operator |
> | compressed: false |
> | Statistics: Num rows: 100 Data size: 14200 Basic stats: COMPLETE Column
> stats: NONE |
> | table: |
> | input format: org.apache.hadoop.mapred.SequenceFileInputFormat |
> | output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat |
> | serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe |
> | |
> | Stage: Stage-0 |
> | Fetch Operator |
> | limit: 100 |
> | Processor Tree: |
> | ListSink |
> | |
> ++--+
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21336) HMS Index PCS_STATS_IDX too long for Oracle when NLS_LENGTH_SEMANTICS=char
[
https://issues.apache.org/jira/browse/HIVE-21336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790128#comment-16790128
]
Hive QA commented on HIVE-21336:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962018/HIVE-21336.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16455/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16455/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16455/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962018 - PreCommit-HIVE-Build
> HMS Index PCS_STATS_IDX too long for Oracle when NLS_LENGTH_SEMANTICS=char
> --
>
> Key: HIVE-21336
> URL: https://issues.apache.org/jira/browse/HIVE-21336
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21336.2.patch, HIVE-21336.3.patch,
> HIVE-21336.4.patch, HIVE-21336.patch
>
>
> CREATE INDEX PCS_STATS_IDX ON PAR T_COL_STATS
> (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME)
> Error: ORA-01450: maximum key length (6398) exceeded (state=72000,code=1450)
> Customer tried the same DDL in SQLDevloper, and got the same error. This
> could be a result of combination of DB level settings like the db_block_size,
> limiting the maximum key length, as per below doc:
> http://www.dba-oracle.com/t_ora_01450_maximum_key_length_exceeded.htm
> Also {{NLS_LENGTH_SEMANTICS}} is by default BYTE, but users can set this at
> the session level to CHAR, thus reducing the max size of the index length. We
> have increased the size of the COLUMN_NAME from 128 to 767 (used to be at
> 1000) and TABLE_NAME from 128 to 256. This by setting
> {code}
> CREATE TABLE PART_COL_STATS (
> CS_ID NUMBER NOT NULL,
> DB_NAME VARCHAR2(128) NOT NULL,
> TABLE_NAME VARCHAR2(256) NOT NULL,
> PARTITION_NAME VARCHAR2(767) NOT NULL,
> COLUMN_NAME VARCHAR2(767) NOT NULL,
> CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS
> (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME);
> {code}
> Reproducer:
> {code}
> SQL*Plus: Release 11.2.0.2.0 Production on Wed Feb 27 11:02:16 2019 Copyright
> (c) 1982, 2011, Oracle. All rights reserved.
> Connected to: Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit
> Production
> SQL> select * from v$nls_parameters where parameter = 'NLS_LENGTH_SEMANTICS';
> PARAMETER
>
> VALUE
>
> NLS_LENGTH_SEMANTICS
> BYTE
> SQL> alter session set NLS_LENGTH_SEMANTICS=CHAR; Session altered.
> SQL> commit; Commit complete.
> SQL> select * from v$nls_parameters where parameter = 'NLS_LENGTH_SEMANTICS';
> PARAMETER
>
> VALUE
>
> NLS_LENGTH_SEMANTICS
> CHAR
> SQL> CREATE TABLE PART_COL_STATS (CS_ID NUMBER NOT NULL, DB_NAME
> VARCHAR2(128) NOT NULL, TABLE_NAME VARCHAR2(256) NOT NULL, PARTITION_NAME
> VARCHAR2(767) NOT NULL, COLUMN_NAME VARCHAR2(767) NOT NULL);
> Table created.
> SQL> CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS
> (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME);
> CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS
> (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME)
> * ERROR at line 1: ORA-01450: maximum key length (6398) exceeded
> SQL> alter session set NLS_LENGTH_SEMANTICS=BYTE;
> Session altered.
> SQL> commit;
> Commit complete.
> SQL> drop table PART_COL_STATS;
> Table dropped.
> SQL> commit;
> Commit complete.
> SQL> CREATE TABLE PART_COL_STATS (CS_ID NUMBER NOT NULL, DB_NAME
> VARCHAR2(128) NOT NULL, TABLE_NAME VARCHAR2(256) NOT NULL, PARTITION_NAME
> VARCHAR2(767) NOT NULL, COLUMN_NAME VARCHAR2(767) NOT NULL);
> Table created.
> SQL> CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS
> (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME);
> Index created.
> SQL> commit;
> Commit complete.
> SQL>
> {code}
--
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Patch Available (was: Open)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch, HIVE-21382.3.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Status: Open (was: Patch Available)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch, HIVE-21382.3.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21382) Group by keys reduction optimization - keys are not reduced in query23
[
https://issues.apache.org/jira/browse/HIVE-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21382:
---
Attachment: (was: HIVE-21382.3.patch)
> Group by keys reduction optimization - keys are not reduced in query23
> --
>
> Key: HIVE-21382
> URL: https://issues.apache.org/jira/browse/HIVE-21382
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21382.1.patch, HIVE-21382.2.patch,
> HIVE-21382.2.patch, HIVE-21382.3.patch
>
>
> {code:sql}
> explain cbo with frequent_ss_items as
> (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date
> solddate,count(*) cnt
> from store_sales
> ,date_dim
> ,item
> where ss_sold_date_sk = d_date_sk
> and ss_item_sk = i_item_sk
> and d_year in (1999,1999+1,1999+2,1999+3)
> group by substr(i_item_desc,1,30),i_item_sk,d_date
> having count(*) >4)
> select sum(sales)
> from ((select cs_quantity*cs_list_price sales
>from catalog_sales
>,date_dim
>where d_year = 1999
> and d_moy = 1
> and cs_sold_date_sk = d_date_sk
> and cs_item_sk in (select item_sk from frequent_ss_items))) subq
> limit 100;
> {code}
> {code:sql}
> HiveSortLimit(fetch=[100])
> HiveProject($f0=[$0])
> HiveAggregate(group=[{}], agg#0=[sum($0)])
> HiveProject(sales=[*(CAST($2):DECIMAL(10, 0), $3)])
> HiveSemiJoin(condition=[=($1, $5)], joinType=[inner])
> HiveJoin(condition=[=($0, $4)], joinType=[inner], algorithm=[none],
> cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(cs_sold_date_sk=[$0], cs_item_sk=[$15],
> cs_quantity=[$18], cs_list_price=[$20])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints, catalog_sales]],
> table:alias=[catalog_sales])
> HiveProject(d_date_sk=[$0])
> HiveFilter(condition=[AND(=($6, 1999), =($8, 1))])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$1])
> HiveFilter(condition=[>($3, 4)])
> HiveProject(substr=[$2], i_item_sk=[$1], d_date=[$0], $f3=[$3])
> HiveAggregate(group=[{3, 4, 5}], agg#0=[count()])
> HiveJoin(condition=[=($1, $4)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveJoin(condition=[=($0, $2)], joinType=[inner],
> algorithm=[none], cost=[{2.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(ss_sold_date_sk=[$0], ss_item_sk=[$2])
> HiveFilter(condition=[IS NOT NULL($0)])
> HiveTableScan(table=[[perf_constraints,
> store_sales]], table:alias=[store_sales])
> HiveProject(d_date_sk=[$0], d_date=[$2])
> HiveFilter(condition=[IN($6, 1999, 2000, 2001, 2002)])
> HiveTableScan(table=[[perf_constraints, date_dim]],
> table:alias=[date_dim])
> HiveProject(i_item_sk=[$0], substr=[substr($4, 1, 30)])
> HiveTableScan(table=[[perf_constraints, item]],
> table:alias=[item])
> {code}
> Right side of HiveSemiJoin has an aggregate which could be reduce to have
> only {{i_item_sk}} as group by key since {{i_item_sk}} is primary key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21338) Remove order by and limit for aggregates
[
https://issues.apache.org/jira/browse/HIVE-21338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790194#comment-16790194
]
Hive QA commented on HIVE-21338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 171 unchanged - 0
fixed = 173 total (was 171) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16457/dev-support/hive-personality.sh
|
| git revision | master / 9f2f101 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16457/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16457/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove order by and limit for aggregates
>
>
> Key: HIVE-21338
> URL: https://issues.apache.org/jira/browse/HIVE-21338
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21338.1.patch, HIVE-21338.2.patch,
> HIVE-21338.3.patch, HIVE-21338.4.patch, HIVE-21338.5.patch,
> HIVE-21338.6.patch, HIVE-21338.7.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If a query is guaranteed to produce at most one row LIMIT and ORDER BY could
> be removed. This saves unnecessary vertex for LIMIT/ORDER BY.
> {code:sql}
> explain select count(*) cs from store_sales where ss_ext_sales_price > 100.00
> order by cs limit 100
> {code}
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: vgarg_20190227131959_2914830f-eab6-425d-b9f0-b8cb56f8a1e9:4
> Edges:
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> DagName: vgarg_20190227131959_2914830f-eab6-425d-b9f0-b8cb56f8a1e9:4
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: store_sales
> filterExpr: (ss_ext_sales_price > 100) (type: boolean)
> Statistics: Num rows: 1 Data size: 112 Basic stats:
> COMPLETE Column stats: NONE
[jira] [Commented] (HIVE-21338) Remove order by and limit for aggregates
[
https://issues.apache.org/jira/browse/HIVE-21338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790222#comment-16790222
]
Hive QA commented on HIVE-21338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962033/HIVE-21338.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962033 - PreCommit-HIVE-Build
> Remove order by and limit for aggregates
>
>
> Key: HIVE-21338
> URL: https://issues.apache.org/jira/browse/HIVE-21338
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21338.1.patch, HIVE-21338.2.patch,
> HIVE-21338.3.patch, HIVE-21338.4.patch, HIVE-21338.5.patch,
> HIVE-21338.6.patch, HIVE-21338.7.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> If a query is guaranteed to produce at most one row LIMIT and ORDER BY could
> be removed. This saves unnecessary vertex for LIMIT/ORDER BY.
> {code:sql}
> explain select count(*) cs from store_sales where ss_ext_sales_price > 100.00
> order by cs limit 100
> {code}
> {code}
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: vgarg_20190227131959_2914830f-eab6-425d-b9f0-b8cb56f8a1e9:4
> Edges:
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
> DagName: vgarg_20190227131959_2914830f-eab6-425d-b9f0-b8cb56f8a1e9:4
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: store_sales
> filterExpr: (ss_ext_sales_price > 100) (type: boolean)
> Statistics: Num rows: 1 Data size: 112 Basic stats:
> COMPLETE Column stats: NONE
> Filter Operator
> predicate: (ss_ext_sales_price > 100) (type: boolean)
> Statistics: Num rows: 1 Data size: 112 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> Statistics: Num rows: 1 Data size: 112 Basic stats:
> COMPLETE Column stats: NONE
> Group By Operator
> aggregations: count()
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 120 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 120 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: bigint)
> Execution mode: vectorized
> Reducer 2
> Execution mode: vectorized
> Reduce Operator Tree:
> Group By Operator
> aggregations: count(VALUE._col0)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: _col0 (type: bigint)
> sort order: +
> Statistics: Num rows: 1 Data size: 120 Basic stats:
> COMPLETE Column stats: NONE
> TopN Hash Memory Usage: 0.1
> Reducer 3
> Execution mode: vectorized
> Reduce Operator Tree:
> Select Operator
> expressions: KEY.reducesinkkey0 (type: bigint)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE
> Column stats: NONE
> Limit
> Number of rows: 100
> Statistics: Num rows: 1 Data size: 120 Basic stats:
> COMPLETE
[jira] [Work logged] (HIVE-20057) For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute change not reflecting for non-CAPS
[
https://issues.apache.org/jira/browse/HIVE-20057?focusedWorklogId=210859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210859
]
ASF GitHub Bot logged work on HIVE-20057:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 06:56
Start Date: 11/Mar/19 06:56
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #388: HIVE-20057: Fix
Hive table conversion DESCRIBE table bug
URL: https://github.com/apache/hive/pull/388#discussion_r264106935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1802,12 +1802,12 @@ private MTable convertToMTable(Table tbl) throws
InvalidObjectException,
// accordingly
String tableType = tbl.getTableType();
boolean isExternal =
Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
-if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
+if (TableType.MANAGED_TABLE.toString().equalsIgnoreCase(tableType)) {
Review comment:
@animenon
Thanks for thr reply! But, I'm still confused.
As per your point, isExternal is proper (both "True" and "TRUE"). But, why
would tableType have different case based on input for EXTERNAL property?
Below is the code that reads EXTERNAL property from ALTER TABLE command and
sets correct tableType.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L4092
```
String externalProp = alterTbl.getProps().get("EXTERNAL");
if (externalProp != null) {
if (Boolean.parseBoolean(externalProp) && tbl.getTableType() ==
TableType.MANAGED_TABLE) {
tbl.setTableType(TableType.EXTERNAL_TABLE);
} else if (!Boolean.parseBoolean(externalProp) && tbl.getTableType()
== TableType.EXTERNAL_TABLE) {
tbl.setTableType(TableType.MANAGED_TABLE);
}
}
```
If you notice, we use Boolean.parseBoolean(externalProp) here which ignores
case.
I didn't get how two values of externalProp ("True" and "TRUE") influences
the table type set via setTableType.
Pls point to what I'm missing here.
@rmsmani
Please check if I miss anything.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210859)
Time Spent: 2h 50m (was: 2h 40m)
> For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute
> change not reflecting for non-CAPS
>
>
> Key: HIVE-20057
> URL: https://issues.apache.org/jira/browse/HIVE-20057
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: All Versions
>Reporter: Anirudh
>Assignee: Anirudh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: hive20057.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Hive EXTERNAL table shown as MANAGED after conversion using
> {code} ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='True')
> {code}
>
> The DESCRIBE FORMATTED shows:
> {code}
> Table Type: MANAGED_TABLE
> Table Parameters:
> EXTERNAL True
> {code}
>
> This is actually a External table but is shown wrongly, as 'True' was used in
> place of 'TRUE' in the ALTER statement.
> Issue explained here:
> [StakOverflow - Hive Table is MANAGED or
> EXTERNAL|https://stackoverflow.com/questions/51103317/hive-table-is-managed-or-external/51142873#51142873]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21419) Partition Pruning not happening when using Apache Ranger masking
[ https://issues.apache.org/jira/browse/HIVE-21419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789184#comment-16789184 ] Gopal V commented on HIVE-21419: bq. the pruning expression is not being set when the masking rule is enabled. Depends on the masking function - what function is used here? Masking functions are marked as non-deterministic in some cases and won't allow partition pruning through them. > Partition Pruning not happening when using Apache Ranger masking > > > Key: HIVE-21419 > URL: https://issues.apache.org/jira/browse/HIVE-21419 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Query Planning >Affects Versions: 2.3.2 > Environment: I used an AWS Cloudformation script from AWS's big data > blog[1]. The EMR AMI uses Hive 2.3.3 and Apache Ranger 1.0.0. > Source Table: > CREATE EXTERNAL TABLE analyst1.lineitem_partitioned ( > `l_orderkey` int, > `l_partkey` int, > `l_suppkey` int, > `l_linenumber` int, > `l_quantity` double, > `l_extendedprice` double, > `l_discount` double, > `l_tax` double, > `l_returnflag` string, > `l_linestatus` string, > `l_commitdate` string, > `l_receiptdate` string, > `l_shipinstruct` string, > `l_shipmode` string, > `l_comment` string > ) PARTITIONED BY (`l_shipdate` string) > STORED AS PARQUET > LOCATION '/user/analyst1/tpch/sf100/lineitem'; > Destination Table: > CREATE EXTERNAL TABLE analyst1.test1( > l_commitdate string, > l_receiptdate string > ) PARTITIONED BY (`l_shipdate` string) > STORED AS PARQUET > LOCATION '/user/analyst1/tpch/sf100/lineitem_parq_partitioned'; > Query: > insert overwrite table analyst1.test1 PARTITION (l_shipdate) > select l_commitdate, l_receiptdate, l_shipdate > from default.lineitem_parq_partitioned > where l_shipdate = '1992-01-02'; > Ranger Masking Rule: > Hive Database: analyst1 > Hive Table: lineitem_partitioned > Mask Condition Option: Custom: "XX" (replace the column with a static > string for simplicity, but our use case uses a complex UDF). > [1] > https://aws.amazon.com/blogs/big-data/implementing-authorization-and-auditing-using-apache-ranger-on-amazon-emr/ > >Reporter: Mert Hocanin >Priority: Major > Attachments: Operators-in-debugger-with-masking.png, > Operators-in-debugger-without-masking.png, hive-jira-schema-explain-plan.txt > > > I have a partitioned table, which I have a Ranger masking policy on a > non-partition column. When I am attempting to query the table that includes > the column that has masking enabled, then partition pruning no longer occurs. > To reproduce: > Create two partitioned tables. I used TPC-H tables as they are publicly > available and will provide the schemas and queries I used. Insert into the > second table from the first table. For example: > insert overwrite table analyst1.test1 PARTITION (l_shipdate) > select l_commitdate, l_receiptdate, l_shipdate > from analyst1.lineitem_partitioned > where l_shipdate = '1992-01-02'; > I have attached the explain plan when a masking rule on l_commitdate is > enabled and when not enabled. > I have done a bit of deep dive and see that the pruning expression is not > being set when the masking rule is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21371) Make NonSyncByteArrayOutputStream Overflow Conscious
[
https://issues.apache.org/jira/browse/HIVE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-21371:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~belugabehr]
> Make NonSyncByteArrayOutputStream Overflow Conscious
> -
>
> Key: HIVE-21371
> URL: https://issues.apache.org/jira/browse/HIVE-21371
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21371.1.patch, HIVE-21371.2.patch,
> HIVE-21371.2.patch, HIVE-21371.2.patch
>
>
> {code:java|title=NonSyncByteArrayOutputStream}
> private int enLargeBuffer(int increment) {
> int temp = count + increment;
> int newLen = temp;
> if (temp > buf.length) {
> if ((buf.length << 1) > temp) {
> newLen = buf.length << 1;
> }
> byte newbuf[] = new byte[newLen];
> System.arraycopy(buf, 0, newbuf, 0, count);
> buf = newbuf;
> }
> return newLen;
> }
> {code}
> This will fail if the array is 2GB or larger because it will double the size
> every time without consideration for the 4GB limit on arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-20057) For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute change not reflecting for non-CAPS
[
https://issues.apache.org/jira/browse/HIVE-20057?focusedWorklogId=210858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210858
]
ASF GitHub Bot logged work on HIVE-20057:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 06:55
Start Date: 11/Mar/19 06:55
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #388: HIVE-20057: Fix
Hive table conversion DESCRIBE table bug
URL: https://github.com/apache/hive/pull/388#discussion_r264106935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1802,12 +1802,12 @@ private MTable convertToMTable(Table tbl) throws
InvalidObjectException,
// accordingly
String tableType = tbl.getTableType();
boolean isExternal =
Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
-if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
+if (TableType.MANAGED_TABLE.toString().equalsIgnoreCase(tableType)) {
Review comment:
@animenon
Thanks for thr reply! But, I'm still confused.
As per your point, isExternal is proper (both "True" and "TRUE"). But, why
would tableType have different case based on input for EXTERNAL property?
Below is the code that reads EXTERNAL property from ALTER TABLE command and
sets correct tableType.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L4092
```
String externalProp = alterTbl.getProps().get("EXTERNAL");
if (externalProp != null) {
if (Boolean.parseBoolean(externalProp) && tbl.getTableType() ==
TableType.MANAGED_TABLE) {
tbl.setTableType(TableType.EXTERNAL_TABLE);
} else if (!Boolean.parseBoolean(externalProp) && tbl.getTableType()
== TableType.EXTERNAL_TABLE) {
tbl.setTableType(TableType.MANAGED_TABLE);
}
}
```
If you notice, we use Boolean.parseBoolean(externalProp) here which ignores
case.
I didn't get how two values of externalProp ("True" and "TRUE") influences
the table type set via setTableType.
Pls point to what I'm missing here.
@rmsmani
Please check if I miss anything.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210858)
Time Spent: 2h 40m (was: 2.5h)
> For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute
> change not reflecting for non-CAPS
>
>
> Key: HIVE-20057
> URL: https://issues.apache.org/jira/browse/HIVE-20057
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: All Versions
>Reporter: Anirudh
>Assignee: Anirudh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: hive20057.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Hive EXTERNAL table shown as MANAGED after conversion using
> {code} ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='True')
> {code}
>
> The DESCRIBE FORMATTED shows:
> {code}
> Table Type: MANAGED_TABLE
> Table Parameters:
> EXTERNAL True
> {code}
>
> This is actually a External table but is shown wrongly, as 'True' was used in
> place of 'TRUE' in the ALTER statement.
> Issue explained here:
> [StakOverflow - Hive Table is MANAGED or
> EXTERNAL|https://stackoverflow.com/questions/51103317/hive-table-is-managed-or-external/51142873#51142873]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: HIVE-21401.10.patch > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[
https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789361#comment-16789361
]
Hive QA commented on HIVE-21325:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961930/HIVE-21325.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15824 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16440/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16440/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16440/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961930 - PreCommit-HIVE-Build
> Hive external table replication failed with Permission denied issue.
>
>
> Key: HIVE-21325
> URL: https://issues.apache.org/jira/browse/HIVE-21325
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch,
> HIVE-21325.03.patch, HIVE-21325.04.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> During external table replication the file copy is done in parallel to the
> meta data replication. If the file copy task creates the directory with do as
> set to true, it will create the directory with permission set to the user
> running the repl command. In that case the meta data task while creating the
> table may fail as hive user might not have access to the created directory.
> The fix should be
> # While creating directory, if sql based authentication is enabled, then
> disable storage based authentication for hive user.
> # Currently the created directory has the login user access, it should
> retain the source clusters owner, group and permission.
> # For external table replication don't create the directory during create
> table and add partition.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21412: -- Attachment: HIVE-21412.2.patch > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21412.1.patch, HIVE-21412.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Status: Open (was: Patch Available) > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch, HIVE-21325.04.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Status: Patch Available (was: Open) > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch, HIVE-21325.04.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789295#comment-16789295
]
Hive QA commented on HIVE-21368:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961927/HIVE-21368.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15826 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=162)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=137)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16439/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16439/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16439/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961927 - PreCommit-HIVE-Build
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-21368.1.patch
>
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21412: -- Comment: was deleted (was: PR: https://github.com/apache/orc/pull/373) > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21412.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21412: -- Status: Patch Available (was: Open) > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21412.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-21368:
--
Attachment: HIVE-21368.1.patch
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-21368.1.patch
>
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Teddy Choi updated HIVE-21368:
--
Status: Patch Available (was: Open)
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-21368.1.patch
>
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789268#comment-16789268
]
Hive QA commented on HIVE-21368:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16439/dev-support/hive-personality.sh
|
| git revision | master / c3939da |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16439/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-21368.1.patch
>
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
>
[jira] [Updated] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21325: --- Attachment: HIVE-21325.04.patch > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch, HIVE-21325.04.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21292) Break up DDLTask - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?focusedWorklogId=210904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210904
]
ASF GitHub Bot logged work on HIVE-21292:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 09:39
Start Date: 11/Mar/19 09:39
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #543:
HIVE-21292: Break up DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264150401
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/DDLTask2.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.hive.ql.CompilationOpContext;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryPlan;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.parse.ExplainConfiguration.AnalyzeState;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+
+/**
+ * DDLTask implementation.
+**/
+public class DDLTask2 extends Task implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private static final Map, Class>> DESC_TO_OPARATION =
+ new HashMap<>();
+ public static void registerOperator(Class descClass,
+ Class> operationClass) {
+DESC_TO_OPARATION.put(descClass, operationClass);
+ }
+
+ @Override
+ public boolean requireLock() {
+return this.work != null && this.work.getNeedLock();
+ }
+
+ @Override
+ public void initialize(QueryState queryState, QueryPlan queryPlan,
DriverContext ctx,
+ CompilationOpContext opContext) {
+super.initialize(queryState, queryPlan, ctx, opContext);
+ }
+
+ @Override
+ public int execute(DriverContext driverContext) {
+if (driverContext.getCtx().getExplainAnalyze() == AnalyzeState.RUNNING) {
+ return 0;
+}
+
+try {
+ Hive db = Hive.get(conf);
+ DDLDesc ddlDesc = work.getDDLDesc();
+
+ if (DESC_TO_OPARATION.containsKey(ddlDesc.getClass())) {
+DDLOperation ddlOperation =
DESC_TO_OPARATION.get(ddlDesc.getClass()).newInstance();
+ddlOperation.init(db, conf, driverContext, ddlDesc);
+return ddlOperation.execute();
+ } else {
+throw new IllegalArgumentException("Unknown DDL request: " +
ddlDesc.getClass());
+ }
+} catch (Throwable e) {
+ failed(e);
+ return 1;
Review comment:
Not all of them, any DDLOperation may return with an error code. Only those
cases when there is an exception is thrown we translate it to 1, as we have no
better option.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210904)
Time Spent: 7h 40m (was: 7.5h)
> Break up DDLTask - extract Database related operations
> --
>
> Key: HIVE-21292
> URL: https://issues.apache.org/jira/browse/HIVE-21292
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch,
> HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch,
> HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch,
> HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch,
> HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch,
>
[jira] [Updated] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21412: -- Attachment: HIVE-21412.1.patch > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21412.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[
https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789419#comment-16789419
]
Hive QA commented on HIVE-21412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961937/HIVE-21412.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16441/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16441/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961937 - PreCommit-HIVE-Build
> PostExecOrcFileDump doesn't work with ACID tables
> -
>
> Key: HIVE-21412
> URL: https://issues.apache.org/jira/browse/HIVE-21412
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21412.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21413) Bad result using UNION using same table
[
https://issues.apache.org/jira/browse/HIVE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789425#comment-16789425
]
Stéphane commented on HIVE-21413:
-
Yes, I confirm.
Using
{quote}set hive.optimize.ppd=false;
{quote}
The result is correct.
Thanks !
I really need to migrate to the latest version ! ;)
> Bad result using UNION using same table
> ---
>
> Key: HIVE-21413
> URL: https://issues.apache.org/jira/browse/HIVE-21413
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 1.2.1
> Environment: engine = mr
> TBLPROPERTIES(
> 'auto.purge'='true',
> 'orc.compress'='ZLIB',
> 'serialization.null.format'=''
> )
>
>Reporter: Stéphane
>Priority: Minor
>
> Hi,
> I have a table named "sau' with 1000 rows.
> The table structure is :
>
> ||col_name||data_type||comment||
> |id|string| |
> |mnta|int| |
> |mntb|int| |
> {quote}select id, 'A' as info, mntA as mnt from sau
> union
> select id, 'B' as info, mntB as mnt from sau
> {quote}
> ==> return only rows with "info=B"
> {quote}with
> cte_union as (
> select id, 'A' as info, mntA as mnt from sau
> union
> select id, 'B' as info, mntB as mnt from sau
> )
> select
> count(*)
> from
> cte_union
> {quote}
> ==> return 1000 but should be 2000
>
> I don't know if this bug is fixed on new version.
> Using "union all" clause, the result is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21420) DescDatabaseOperation should not handle test use cases
[ https://issues.apache.org/jira/browse/HIVE-21420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21420: -- Description: [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have testswe should have some other way to address this * database properties should be in alphabetical order, even if the desc operation is not executed in a test * location should be set to a specific value for test cases somewhere else was:[~kgyrtkirk]: DescDatabaseOperation should not be customized because we have testswe should have some other way to address this > DescDatabaseOperation should not handle test use cases > --- > > Key: HIVE-21420 > URL: https://issues.apache.org/jira/browse/HIVE-21420 > Project: Hive > Issue Type: Improvement >Reporter: Miklos Gergely >Priority: Major > > [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have > testswe should have some other way to address this > * database properties should be in alphabetical order, even if the desc > operation is not executed in a test > * location should be set to a specific value for test cases somewhere else -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21292) Break up DDLTask - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?focusedWorklogId=210905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210905
]
ASF GitHub Bot logged work on HIVE-21292:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 09:42
Start Date: 11/Mar/19 09:42
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #543:
HIVE-21292: Break up DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264151355
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/CreateDatabaseOperation.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState;
+
+/**
+ * Operation process of creating a database.
+ */
+public class CreateDatabaseOperation extends DDLOperation {
+ private static final String DATABASE_PATH_SUFFIX = ".db";
+
+ @Override
+ public int execute() throws HiveException {
+Database database = new Database();
+database.setName(ddlDesc.getName());
+database.setDescription(ddlDesc.getComment());
+database.setLocationUri(ddlDesc.getLocationUri());
+database.setParameters(ddlDesc.getDatabaseProperties());
+database.setOwnerName(SessionState.getUserFromAuthenticator());
+database.setOwnerType(PrincipalType.USER);
+
+try {
+ makeLocationQualified(database);
+ db.createDatabase(database, ddlDesc.getIfNotExists());
+} catch (AlreadyExistsException ex) {
+ //it would be better if AlreadyExistsException had an errorCode field
+ throw new HiveException(ex, ErrorMsg.DATABASE_ALREADY_EXISTS,
ddlDesc.getName());
+}
+
+return 0;
+ }
+
+ private void makeLocationQualified(Database database) throws HiveException {
Review comment:
I don't see how, the goal of this method is to make the location uri of the
database qualified, so Database needs to be passed to it. Or would you rather
have a method that just returns the new qualified locationUri, and then pass it
to the database?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210905)
Time Spent: 7h 50m (was: 7h 40m)
> Break up DDLTask - extract Database related operations
> --
>
> Key: HIVE-21292
> URL: https://issues.apache.org/jira/browse/HIVE-21292
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch,
> HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch,
> HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch,
> HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch,
> HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch,
> HIVE-21292.15.patch, HIVE-21292.15.patch, HIVE-21292.16.patch,
> HIVE-21292.17.patch
>
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> DDLTask is a huge class, more than 5000 lines long. The related
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789386#comment-16789386 ] Sankar Hariappan commented on HIVE-21325: - +1 04.patch LGTM > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch, HIVE-21325.04.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21030: -- Attachment: HIVE-21030.5.branch-3.patch > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.branch-3.patch, > HIVE-21030.2.patch, HIVE-21030.3.branch-3.patch, HIVE-21030.3.patch, > HIVE-21030.4.branch-3.patch, HIVE-21030.4.patch, HIVE-21030.5.branch-3.patch, > HIVE-21030.5.patch, HIVE-21030.6.patch, HIVE-21030.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?focusedWorklogId=210861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210861 ] ASF GitHub Bot logged work on HIVE-21286: - Author: ASF GitHub Bot Created on: 11/Mar/19 06:59 Start Date: 11/Mar/19 06:59 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #551: HIVE-21286: Hive should support clean-up of previously bootstrapped tables when retry from different dump. URL: https://github.com/apache/hive/pull/551 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 210861) Time Spent: 6h (was: 5h 50m) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21286.01.patch, HIVE-21286.02.patch, > HIVE-21286.03.patch, HIVE-21286.04.patch, HIVE-21286.05.patch > > Time Spent: 6h > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.clean.tables.from.bootstrap= > Hive will use this config only if the current dump is combined bootstrap in > incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21292) Break up DDLTask - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?focusedWorklogId=210893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210893
]
ASF GitHub Bot logged work on HIVE-21292:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 09:04
Start Date: 11/Mar/19 09:04
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #543:
HIVE-21292: Break up DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264137620
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
+ }
+
+ String location = database.getLocationUri();
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST)) {
+location = "location/in/test";
Review comment:
HIVE-21420 opened
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210893)
Time Spent: 7h 20m (was: 7h 10m)
> Break up DDLTask - extract Database related operations
> --
>
> Key: HIVE-21292
> URL: https://issues.apache.org/jira/browse/HIVE-21292
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch,
> HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch,
> HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch,
> HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch,
> HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch,
> HIVE-21292.15.patch, HIVE-21292.15.patch, HIVE-21292.16.patch,
> HIVE-21292.17.patch
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789437#comment-16789437
]
Peter Vary commented on HIVE-21402:
---
I agree with [~ashutoshc] that this is most probably a config issue. I usually
use the same hive-site.xml for HMS and HS2. This might caused the problem.
There relevant configs are:
{code:java}
hive.txn.managerorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager
hive.support.concurrencytrue
hive.compactor.initiator.ontrue
hive.compactor.worker.threads5
hive.compactor.crud.query.basedtrue{code}
On the other hand this highlights the issue that we can have other Throwables
in that try-catch which might prevent setting the correct state of the
compaction. So I think this change should go in anyway. What do you think?
Also - maybe in a follow-up Jira - it would be good to store the reason for the
failure for the failed transactions. This could help troubleshooting
tremendously. [~vgumashta]: Is {{CompactionInfo.metaInfo}} is designed to store
info about the compaction - could it be used to store the exception message for
the failed compactions?
Thanks,
Peter
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
[
https://issues.apache.org/jira/browse/HIVE-21368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789239#comment-16789239
]
Teddy Choi commented on HIVE-21368:
---
[~gopalv], thanks for pointing out. I made a patch for it.
> Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion
> --
>
> Key: HIVE-21368
> URL: https://issues.apache.org/jira/browse/HIVE-21368
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-21368.1.patch
>
>
> Joins projecting Decimal64 have a suspicious cast in the inner loop
> {code}
> ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
> {code}
> {code}
> create temporary table foo(x int , y decimal(7,2));
> create temporary table bar(x int , y decimal(7,2));
> set hive.explain.user=false;
> explain vectorization detail select sum(foo.y) from foo, bar where foo.x =
> bar.x;
> {code}
> {code}
> ' Map Join Operator'
> 'condition map:'
> ' Inner Join 0 to 1'
> 'keys:'
> ' 0 _col0 (type: int)'
> ' 1 _col0 (type: int)'
> 'Map Join Vectorization:'
> 'bigTableKeyColumnNums: [0]'
> 'bigTableRetainedColumnNums: [3]'
> 'bigTableValueColumnNums: [3]'
> 'bigTableValueExpressions:
> ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
> 'className: VectorMapJoinInnerBigOnlyLongOperator'
> 'native: true'
> 'nativeConditionsMet:
> hive.mapjoin.optimized.hashtable IS true,
> hive.vectorized.execution.mapjoin.native.enabled IS true,
> hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS
> true, No nullsafe IS true, Small table vectorizes IS true, Fast Hash Table
> and No Hybrid Hash Join IS true'
> 'projectedOutputColumnNums: [3]'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789284#comment-16789284
]
Denys Kuzmenko commented on HIVE-21397:
---
[~gopalv] could you please check once again, I have done some refactoring.
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: leishuiyu
>Priority: Blocker
> Attachments: OrcUtils.patch, orc_file_dump.out, orc_file_dump.q
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8
[jira] [Work logged] (HIVE-21292) Break up DDLTask - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?focusedWorklogId=210892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210892
]
ASF GitHub Bot logged work on HIVE-21292:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 09:04
Start Date: 11/Mar/19 09:04
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #543:
HIVE-21292: Break up DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264137508
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
Review comment:
HIVE-21420 opened
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210892)
Time Spent: 7h 10m (was: 7h)
> Break up DDLTask - extract Database related operations
> --
>
> Key: HIVE-21292
> URL: https://issues.apache.org/jira/browse/HIVE-21292
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch,
> HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch,
> HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch,
> HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch,
> HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch,
> HIVE-21292.15.patch, HIVE-21292.15.patch, HIVE-21292.16.patch,
> HIVE-21292.17.patch
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: (was: HIVE-21401.10.patch) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21292) Break up DDLTask - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?focusedWorklogId=210900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210900
]
ASF GitHub Bot logged work on HIVE-21292:
-
Author: ASF GitHub Bot
Created on: 11/Mar/19 09:35
Start Date: 11/Mar/19 09:35
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #543:
HIVE-21292: Break up DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264148937
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
+ }
+
+ String location = database.getLocationUri();
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST)) {
+location = "location/in/test";
+ }
+
+ PrincipalType ownerType = database.getOwnerType();
+ formatter.showDatabaseDescription(outStream, database.getName(),
database.getDescription(), location,
+ database.getOwnerName(), (null == ownerType) ? null :
ownerType.name(), params);
Review comment:
Fixing in the next refactor in the DDL cleanup.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 210900)
Time Spent: 7.5h (was: 7h 20m)
> Break up DDLTask - extract Database related operations
> --
>
> Key: HIVE-21292
> URL: https://issues.apache.org/jira/browse/HIVE-21292
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch,
> HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch,
> HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch,
> HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch,
> HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch,
> HIVE-21292.15.patch, HIVE-21292.15.patch, HIVE-21292.16.patch,
> HIVE-21292.17.patch
>
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[
https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789335#comment-16789335
]
Hive QA commented on HIVE-21325:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 18 unchanged - 1 fixed
= 19 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16440/dev-support/hive-personality.sh
|
| git revision | master / c3939da |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16440/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore/metastore-server ql itests/hive-unit
itests/hive-unit-hadoop2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16440/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive external table replication failed with Permission denied issue.
>
>
> Key: HIVE-21325
> URL: https://issues.apache.org/jira/browse/HIVE-21325
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch,
> HIVE-21325.03.patch, HIVE-21325.04.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> During external table replication the file copy is done in parallel to the
> meta data replication. If the file copy task creates the
[jira] [Updated] (HIVE-21420) DescDatabaseOperation should not handle test use cases
[ https://issues.apache.org/jira/browse/HIVE-21420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21420: -- Description: [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have testswe should have some other way to address this. Location should be set to a specific value for test cases somewhere else (was: [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have testswe should have some other way to address this * database properties should be in alphabetical order, even if the desc operation is not executed in a test * location should be set to a specific value for test cases somewhere else) > DescDatabaseOperation should not handle test use cases > --- > > Key: HIVE-21420 > URL: https://issues.apache.org/jira/browse/HIVE-21420 > Project: Hive > Issue Type: Improvement >Reporter: Miklos Gergely >Priority: Major > > [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have > testswe should have some other way to address this. Location should be > set to a specific value for test cases somewhere else -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: HIVE-21401.10.patch > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-21397:
-
Assignee: Denys Kuzmenko (was: leishuiyu)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: OrcUtils.patch, orc_file_dump.out, orc_file_dump.q
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent
[jira] [Commented] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[
https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789384#comment-16789384
]
Hive QA commented on HIVE-21412:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16441/dev-support/hive-personality.sh
|
| git revision | master / c3939da |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16441/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PostExecOrcFileDump doesn't work with ACID tables
> -
>
> Key: HIVE-21412
> URL: https://issues.apache.org/jira/browse/HIVE-21412
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21412.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-21413) Bad result using UNION using same table
[
https://issues.apache.org/jira/browse/HIVE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stéphane resolved HIVE-21413.
-
Resolution: Won't Fix
Alternative way to solve the issue.
> Bad result using UNION using same table
> ---
>
> Key: HIVE-21413
> URL: https://issues.apache.org/jira/browse/HIVE-21413
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Affects Versions: 1.2.1
> Environment: engine = mr
> TBLPROPERTIES(
> 'auto.purge'='true',
> 'orc.compress'='ZLIB',
> 'serialization.null.format'=''
> )
>
>Reporter: Stéphane
>Priority: Minor
>
> Hi,
> I have a table named "sau' with 1000 rows.
> The table structure is :
>
> ||col_name||data_type||comment||
> |id|string| |
> |mnta|int| |
> |mntb|int| |
> {quote}select id, 'A' as info, mntA as mnt from sau
> union
> select id, 'B' as info, mntB as mnt from sau
> {quote}
> ==> return only rows with "info=B"
> {quote}with
> cte_union as (
> select id, 'A' as info, mntA as mnt from sau
> union
> select id, 'B' as info, mntB as mnt from sau
> )
> select
> count(*)
> from
> cte_union
> {quote}
> ==> return 1000 but should be 2000
>
> I don't know if this bug is fixed on new version.
> Using "union all" clause, the result is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Patch Available (was: Open) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21423) Do not check for whitespace issues in generated code
[ https://issues.apache.org/jira/browse/HIVE-21423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21423: - > Do not check for whitespace issues in generated code > > > Key: HIVE-21423 > URL: https://issues.apache.org/jira/browse/HIVE-21423 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21423) Do not check for whitespace issues in generated code
[ https://issues.apache.org/jira/browse/HIVE-21423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21423: -- Status: Patch Available (was: Open) > Do not check for whitespace issues in generated code > > > Key: HIVE-21423 > URL: https://issues.apache.org/jira/browse/HIVE-21423 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21423.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21423) Do not check for whitespace issues in generated code
[ https://issues.apache.org/jira/browse/HIVE-21423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21423: -- Attachment: HIVE-21423.01.patch > Do not check for whitespace issues in generated code > > > Key: HIVE-21423 > URL: https://issues.apache.org/jira/browse/HIVE-21423 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21423.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[ https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-21421: --- > HiveStatement.getQueryId throws NPE when query is not running. > --- > > Key: HIVE-21421 > URL: https://issues.apache.org/jira/browse/HIVE-21421 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > > HiveStatement.getQueryId throws NullPointerException if it invoked without > executing any query or query is closed. It should instead return null so that > caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21325) Hive external table replication failed with Permission denied issue.
[ https://issues.apache.org/jira/browse/HIVE-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789452#comment-16789452 ] mahesh kumar behera commented on HIVE-21325: 04.patch pushed to master ..thanks [~sankarh] for review > Hive external table replication failed with Permission denied issue. > > > Key: HIVE-21325 > URL: https://issues.apache.org/jira/browse/HIVE-21325 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21325.01.patch, HIVE-21325.02.patch, > HIVE-21325.03.patch, HIVE-21325.04.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > During external table replication the file copy is done in parallel to the > meta data replication. If the file copy task creates the directory with do as > set to true, it will create the directory with permission set to the user > running the repl command. In that case the meta data task while creating the > table may fail as hive user might not have access to the created directory. > The fix should be > # While creating directory, if sql based authentication is enabled, then > disable storage based authentication for hive user. > # Currently the created directory has the login user access, it should > retain the source clusters owner, group and permission. > # For external table replication don't create the directory during create > table and add partition. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789485#comment-16789485
]
Hive QA commented on HIVE-21030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961948/HIVE-21030.5.branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16443/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16443/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16443/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-11 12:26:34.571
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16443/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-11 12:26:34.573
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
c3939da..4274f48 master -> origin/master
+ git reset --hard HEAD
HEAD is now at c3939da HIVE-21371: Make NonSyncByteArrayOutputStream Overflow
Conscious (David Mollitor, reviewed by Gopal V)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout branch-3
Switched to branch 'branch-3'
Your branch is behind 'origin/branch-3' by 14 commits, and can be
fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/branch-3
HEAD is now at 4fd9425 HIVE-21376: Incompatible change in Hive bucket
computation (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-11 12:26:38.719
+ rm -rf ../yetus_PreCommit-HIVE-Build-16443
+ mkdir ../yetus_PreCommit-HIVE-Build-16443
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16443
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16443/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java: does
not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/ql/exec/TestHiveCredentialProviders.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc7528079441194615186.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc7528079441194615186.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
[ERROR] Failed to execute goal on project hive-shims-common: Could not resolve
dependencies for project
org.apache.hive.shims:hive-shims-common:jar:3.2.0-SNAPSHOT: Could not find
artifact org.apache.httpcomponents:httpcore:jar:4.4.4 in datanucleus
(http://www.datanucleus.org/downloads/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace
[jira] [Work logged] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[ https://issues.apache.org/jira/browse/HIVE-21421?focusedWorklogId=210951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-210951 ] ASF GitHub Bot logged work on HIVE-21421: - Author: ASF GitHub Bot Created on: 11/Mar/19 12:33 Start Date: 11/Mar/19 12:33 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #562: HIVE-21421: HiveStatement.getQueryId throws NPE when query is not running. URL: https://github.com/apache/hive/pull/562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 210951) Time Spent: 10m Remaining Estimate: 0h > HiveStatement.getQueryId throws NPE when query is not running. > --- > > Key: HIVE-21421 > URL: https://issues.apache.org/jira/browse/HIVE-21421 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21421.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > HiveStatement.getQueryId throws NullPointerException if it invoked without > executing any query or query is closed. It should instead return null so that > caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[ https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21421: -- Labels: pull-request-available (was: ) > HiveStatement.getQueryId throws NPE when query is not running. > --- > > Key: HIVE-21421 > URL: https://issues.apache.org/jira/browse/HIVE-21421 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21421.01.patch > > > HiveStatement.getQueryId throws NullPointerException if it invoked without > executing any query or query is closed. It should instead return null so that > caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[ https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21421: Status: Patch Available (was: Open) > HiveStatement.getQueryId throws NPE when query is not running. > --- > > Key: HIVE-21421 > URL: https://issues.apache.org/jira/browse/HIVE-21421 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Attachments: HIVE-21421.01.patch > > > HiveStatement.getQueryId throws NullPointerException if it invoked without > executing any query or query is closed. It should instead return null so that > caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[ https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21421: Attachment: HIVE-21421.01.patch > HiveStatement.getQueryId throws NPE when query is not running. > --- > > Key: HIVE-21421 > URL: https://issues.apache.org/jira/browse/HIVE-21421 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Attachments: HIVE-21421.01.patch > > > HiveStatement.getQueryId throws NullPointerException if it invoked without > executing any query or query is closed. It should instead return null so that > caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Attachment: HIVE-21401.11.patch > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Status: Open (was: Patch Available) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. Also create the new > internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Labels: llap (was: ) Attachment: HIVE-21422.patch Tags: llap Status: Patch Available (was: Open) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[
https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789609#comment-16789609
]
Hive QA commented on HIVE-21421:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} jdbc: The patch generated 1 new + 6 unchanged - 0
fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16445/dev-support/hive-personality.sh
|
| git revision | master / 4274f48 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16445/yetus/diff-checkstyle-jdbc.txt
|
| modules | C: jdbc itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16445/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveStatement.getQueryId throws NPE when query is not running.
> ---
>
> Key: HIVE-21421
> URL: https://issues.apache.org/jira/browse/HIVE-21421
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21421.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HiveStatement.getQueryId throws NullPointerException if it invoked without
> executing any query or query is closed. It should instead return null so that
> caller would check it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21314) Hive Replication not retaining the owner in the replicated table
[ https://issues.apache.org/jira/browse/HIVE-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21314: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Hive Replication not retaining the owner in the replicated table > > > Key: HIVE-21314 > URL: https://issues.apache.org/jira/browse/HIVE-21314 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21314.01.patch, HIVE-21314.02.patch, > HIVE-21314.03.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Hive Replication not retaining the owner in the replicated table. The owner > for the target table is set same as the user executing the load command. The > user information should be read from the dump metadata and should be used > while creating the table at target cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789468#comment-16789468
]
Hive QA commented on HIVE-21401:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
14s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 18 new + 1639 unchanged - 141
fixed = 1657 total (was 1780) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hcatalog/core: The patch generated 1 new + 40
unchanged - 8 fixed = 41 total (was 48) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
23s{color} | {color:red} ql generated 1 new + 2256 unchanged - 2 fixed = 2257
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} accumulo-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} contrib in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} hbase-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} hcatalog/core generated 0 new + 28 unchanged - 1
fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} util in the patch passed. {color} |
|
[jira] [Updated] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21030: -- Attachment: HIVE-21030.6.branch-3.patch > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.branch-3.patch, > HIVE-21030.2.patch, HIVE-21030.3.branch-3.patch, HIVE-21030.3.patch, > HIVE-21030.4.branch-3.patch, HIVE-21030.4.patch, HIVE-21030.5.branch-3.patch, > HIVE-21030.5.patch, HIVE-21030.6.branch-3.patch, HIVE-21030.6.patch, > HIVE-21030.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789580#comment-16789580 ] Ashutosh Chauhan commented on HIVE-21412: - +1 > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21412.1.patch, HIVE-21412.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21422: > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789572#comment-16789572 ] Oliver Draese commented on HIVE-21422: -- Added pull request for this patch: https://github.com/apache/hive/pull/563/files > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[
https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789575#comment-16789575
]
Hive QA commented on HIVE-21412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961956/HIVE-21412.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15824 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16444/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16444/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16444/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961956 - PreCommit-HIVE-Build
> PostExecOrcFileDump doesn't work with ACID tables
> -
>
> Key: HIVE-21412
> URL: https://issues.apache.org/jira/browse/HIVE-21412
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21412.1.patch, HIVE-21412.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789614#comment-16789614
]
Ashutosh Chauhan commented on HIVE-21402:
-
I am unsure of how to deal with unchecked exceptions. IMHO, its not useful to
catch Throwable since in case of unchecked exception its very likely that
compaction will fail in next iteration too, likely that error will be
encountered every time (e.g., was the case here of missing jar). In such cases,
its better to let Throwable escape (or raise InterrruptedException) so that its
dealt with in caller which should then fail the process. For end user its not
useful that HS2 keeps on running where every compaction fails.
On the other hand there is already catch(Throwable) in the outer loop :
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java#L238
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789481#comment-16789481
]
Hive QA commented on HIVE-21401:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961941/HIVE-21401.10.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15825 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16442/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961941 - PreCommit-HIVE-Build
> Break up DDLTask - extract Table related operations
> ---
>
> Key: HIVE-21401
> URL: https://issues.apache.org/jira/browse/HIVE-21401
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch,
> HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch,
> HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch,
> HIVE-21401.09.patch, HIVE-21401.10.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic to the actual operations
> * right now let's ignore the issue of having some operations handled by
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the
> code base the new ones in the new package are called DDLTask2 and DDLWork2
> thus avoiding the usage of fully qualified class names where both the old and
> the new classes are in use.
> Step #2: extract all the table related operations from the old DDLTask except
> alter table, and move them under the new package. Also create the new
> internal framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[
https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789525#comment-16789525
]
Hive QA commented on HIVE-21412:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16444/dev-support/hive-personality.sh
|
| git revision | master / 4274f48 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16444/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PostExecOrcFileDump doesn't work with ACID tables
> -
>
> Key: HIVE-21412
> URL: https://issues.apache.org/jira/browse/HIVE-21412
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21412.1.patch, HIVE-21412.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21412) PostExecOrcFileDump doesn't work with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-21412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-21412: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Denys! > PostExecOrcFileDump doesn't work with ACID tables > - > > Key: HIVE-21412 > URL: https://issues.apache.org/jira/browse/HIVE-21412 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21412.1.patch, HIVE-21412.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21197) Hive replication can add duplicate data during migration to a target with hive.strict.managed.tables enabled
[ https://issues.apache.org/jira/browse/HIVE-21197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21197: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Hive replication can add duplicate data during migration to a target with > hive.strict.managed.tables enabled > > > Key: HIVE-21197 > URL: https://issues.apache.org/jira/browse/HIVE-21197 > Project: Hive > Issue Type: Task > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21197.01.patch, HIVE-21197.02.patch, > HIVE-21197.03.patch, HIVE-21197.04.patch, HIVE-21197.05.patch > > Time Spent: 22h 20m > Remaining Estimate: 0h > > During bootstrap phase it may happen that the files copied to target are > created by events which are not part of the bootstrap. This is because of the > fact that, bootstrap first gets the last event id and then the file list. > During this period if some event are added, then bootstrap will include files > created by these events also.The same files will be copied again during the > first incremental replication just after the bootstrap. In normal scenario, > the duplicate copy does not cause any issue as hive allows the use of target > database only after the first incremental. But in case of migration, the file > at source and target are copied to different location (based on the write id > at target) and thus this may lead to duplicate data at target. This can be > avoided by having at check at load time for duplicate file. This check can be > done only for the first incremental and the search can be done in the > bootstrap directory (with write id 1). if the file is already present then > just ignore the copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21416) Log git apply tries with p0, p1, and p2
[ https://issues.apache.org/jira/browse/HIVE-21416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21416: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you [~mgergely] > Log git apply tries with p0, p1, and p2 > --- > > Key: HIVE-21416 > URL: https://issues.apache.org/jira/browse/HIVE-21416 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21416.01.patch > > > Currently when the PreCommit-HIVE-Build Jenkins job is trying to apply the > patch it tries it first with -p0, then if it wasn't successful with -p1, then > finally if it still wasn't successful with -p2. The 3 tries are not separated > by anything, so the error messages of the potential failures are mixed > together. There should be a log message before each try. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21424) Disable AggregateStatsCache by default
[ https://issues.apache.org/jira/browse/HIVE-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789716#comment-16789716 ] Zoltan Haindrich commented on HIVE-21424: - I think we should either fix aggregate column stats to not "skip" some partitions; or disable it completely... based on Ashutosh's comment; we probably won't be using it in long term: https://issues.apache.org/jira/browse/HIVE-20585?focusedCommentId=16788761=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16788761 this cache right now causes corruption of statistic states in tests randomly, so it would be better to disable it. for example: HIVE-20585 > Disable AggregateStatsCache by default > -- > > Key: HIVE-21424 > URL: https://issues.apache.org/jira/browse/HIVE-21424 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21424.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
[
https://issues.apache.org/jira/browse/HIVE-21421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789663#comment-16789663
]
Hive QA commented on HIVE-21421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961971/HIVE-21421.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16445/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16445/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16445/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961971 - PreCommit-HIVE-Build
> HiveStatement.getQueryId throws NPE when query is not running.
> ---
>
> Key: HIVE-21421
> URL: https://issues.apache.org/jira/browse/HIVE-21421
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21421.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HiveStatement.getQueryId throws NullPointerException if it invoked without
> executing any query or query is closed. It should instead return null so that
> caller would check it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21423) Do not check for whitespace issues in generated code
[ https://issues.apache.org/jira/browse/HIVE-21423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789681#comment-16789681 ] Zoltan Haindrich commented on HIVE-21423: - +1 ; I think this change will only come into effect after updating ptest. > Do not check for whitespace issues in generated code > > > Key: HIVE-21423 > URL: https://issues.apache.org/jira/browse/HIVE-21423 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21423.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789749#comment-16789749
]
Hive QA commented on HIVE-21401:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 18 new + 1639 unchanged - 141
fixed = 1657 total (was 1780) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/core: The patch generated 1 new + 40
unchanged - 8 fixed = 41 total (was 48) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
25s{color} | {color:red} ql generated 1 new + 2256 unchanged - 2 fixed = 2257
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} accumulo-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} contrib in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} hbase-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} hcatalog/core generated 0 new + 28 unchanged - 1
fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} util in the patch passed. {color} |
|
[jira] [Updated] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21109: -- Attachment: HIVE-21109.01.patch Status: Patch Available (was: Open) Initial work to trigger ptests. > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Attachments: HIVE-21109.01.patch > > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789756#comment-16789756
]
Hive QA commented on HIVE-21401:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961975/HIVE-21401.11.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15825 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16446/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16446/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16446/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961975 - PreCommit-HIVE-Build
> Break up DDLTask - extract Table related operations
> ---
>
> Key: HIVE-21401
> URL: https://issues.apache.org/jira/browse/HIVE-21401
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch,
> HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch,
> HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch,
> HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic to the actual operations
> * right now let's ignore the issue of having some operations handled by
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the
> code base the new ones in the new package are called DDLTask2 and DDLWork2
> thus avoiding the usage of fully qualified class names where both the old and
> the new classes are in use.
> Step #2: extract all the table related operations from the old DDLTask except
> alter table, and move them under the new package. Also create the new
> internal framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21424) Disable AggregateStatsCache by default
[ https://issues.apache.org/jira/browse/HIVE-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21424: Status: Patch Available (was: Open) > Disable AggregateStatsCache by default > -- > > Key: HIVE-21424 > URL: https://issues.apache.org/jira/browse/HIVE-21424 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21424.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21424) Disable AggregateStatsCache by default
[ https://issues.apache.org/jira/browse/HIVE-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21424: Attachment: HIVE-21424.01.patch > Disable AggregateStatsCache by default > -- > > Key: HIVE-21424 > URL: https://issues.apache.org/jira/browse/HIVE-21424 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21424.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21424) Disable AggregateStatsCache by default
[ https://issues.apache.org/jira/browse/HIVE-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-21424: --- > Disable AggregateStatsCache by default > -- > > Key: HIVE-21424 > URL: https://issues.apache.org/jira/browse/HIVE-21424 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789768#comment-16789768
]
Hive QA commented on HIVE-21030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-16447/patches/PreCommit-HIVE-Build-16447.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16447/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch, HIVE-21030.2.branch-3.patch,
> HIVE-21030.2.patch, HIVE-21030.3.branch-3.patch, HIVE-21030.3.patch,
> HIVE-21030.4.branch-3.patch, HIVE-21030.4.patch, HIVE-21030.5.branch-3.patch,
> HIVE-21030.5.patch, HIVE-21030.6.branch-3.patch, HIVE-21030.6.patch,
> HIVE-21030.branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789655#comment-16789655
]
Peter Vary commented on HIVE-21402:
---
Yeah, and that catch just prints out the error to the log and leave the
compaction in "working" status. That's left me scratching my head for a while :D
My understanding of the compaction is the following (mostly by documentation
ATM):
* If a compaction fails then it is put to the COMPLETED_COMPACTION table with
the status marked as failed. And will be retried later if the conditions are
still met.
* If the number of the compaction failures are bigger for that compaction than
{{metastore.compactor.initiator.failed.compacts.threshold}} then it will not be
scheduled again.
* If a compaction is found in the "working" state for longer than
{{hive.compactor.worker.timeout}} by the initiator thread then it is put back
to "initiated" state - so it will be queued again later. The config comment
says "declared failed" but I think it does not put a new entry to the
COMPLETED_COMPACTION table, so it is not counted when checking against the
failed.compacts.threshold.
So if my understanding the above process is correct then if we catch the
Throwable then we will have a few (by default 2) failed compactions very close
to each other, on the other hand if we do not catch Throwable then we will have
a continuously "working" compaction forever.
Or maybe I am totally off - learning/learning/learning :) :) :)
Thanks,
Peter
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21424) Disable AggregateStatsCache by default
[ https://issues.apache.org/jira/browse/HIVE-21424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21424: Attachment: HIVE-21424.02.patch > Disable AggregateStatsCache by default > -- > > Key: HIVE-21424 > URL: https://issues.apache.org/jira/browse/HIVE-21424 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21424.01.patch, HIVE-21424.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[ https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789738#comment-16789738 ] Zoltan Haindrich commented on HIVE-21316: - [~ashutoshc] Could you please take a look? > Comparision of varchar column and string literal should happen in varchar > - > > Key: HIVE-21316 > URL: https://issues.apache.org/jira/browse/HIVE-21316 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch, > HIVE-21316.03.patch > > > this is most probably the root cause behind HIVE-21310 as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21425) Use DirectExecutorService for getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21425:
--
Status: Open (was: Patch Available)
> Use DirectExecutorService for getInputSummary
> -
>
> Key: HIVE-21425
> URL: https://issues.apache.org/jira/browse/HIVE-21425
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21425.1.patch, HIVE-21425.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|title=Utilities.java}
> int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
> pathNeedProcess.size());
> if (numExecutors > 1) {
> LOG.info("Using {} threads for getContentSummary", numExecutors);
> executor = Executors.newFixedThreadPool(numExecutors,
> new ThreadFactoryBuilder().setDaemon(true)
> .setNameFormat("Get-Input-Summary-%d").build());
> } else {
> executor = null;
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
> Instead of using a 'null' {{ExecutorService}}, use Guava's
> {{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21425) Use DirectExecutorService for getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21425:
--
Attachment: HIVE-21425.1.patch
> Use DirectExecutorService for getInputSummary
> -
>
> Key: HIVE-21425
> URL: https://issues.apache.org/jira/browse/HIVE-21425
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21425.1.patch, HIVE-21425.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|title=Utilities.java}
> int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
> pathNeedProcess.size());
> if (numExecutors > 1) {
> LOG.info("Using {} threads for getContentSummary", numExecutors);
> executor = Executors.newFixedThreadPool(numExecutors,
> new ThreadFactoryBuilder().setDaemon(true)
> .setNameFormat("Get-Input-Summary-%d").build());
> } else {
> executor = null;
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
> Instead of using a 'null' {{ExecutorService}}, use Guava's
> {{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-335) External Tables should have the option to be marked Read Only
[ https://issues.apache.org/jira/browse/HIVE-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790077#comment-16790077 ] Jim Huang commented on HIVE-335: I am also searching for this feature. It seems the workaround will be more complicated now that Apache Ranger maybe another level of check being enforced. If the dataset in question is small enough and Apache Ranger rules allows data copy, the most naive way is to duplicate that dataset as a workaround to this problem. > External Tables should have the option to be marked Read Only > - > > Key: HIVE-335 > URL: https://issues.apache.org/jira/browse/HIVE-335 > Project: Hive > Issue Type: Improvement > Components: Metastore, Query Processor >Reporter: Richard Lee >Priority: Major > > When creating an External Table, it'd be awesome to have the option of NOT > allowing writes to it (disallow any INSERTs or if hive ever allows UPDATEs). > Adding and Dropping Partitions should still be allowed. > This will enable hive to "play well" with external data stores other than > hdfs where data should be non-maleable. > I'd recomend the following syntax, which applies ONLY to external tables: > CREATE EXTERNAL [READONLY] TABLE ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21425) Use DirectExecutorService for getInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21425:
--
Status: Patch Available (was: Open)
> Use DirectExecutorService for getInputSummary
> -
>
> Key: HIVE-21425
> URL: https://issues.apache.org/jira/browse/HIVE-21425
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-21425.1.patch, HIVE-21425.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java|title=Utilities.java}
> int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
> pathNeedProcess.size());
> if (numExecutors > 1) {
> LOG.info("Using {} threads for getContentSummary", numExecutors);
> executor = Executors.newFixedThreadPool(numExecutors,
> new ThreadFactoryBuilder().setDaemon(true)
> .setNameFormat("Get-Input-Summary-%d").build());
> } else {
> executor = null;
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
> Instead of using a 'null' {{ExecutorService}}, use Guava's
> {{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790100#comment-16790100
]
Hive QA commented on HIVE-21401:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 18 new + 1639 unchanged - 141
fixed = 1657 total (was 1780) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hcatalog/core: The patch generated 1 new + 40
unchanged - 8 fixed = 41 total (was 48) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
20s{color} | {color:red} ql generated 1 new + 2256 unchanged - 2 fixed = 2257
total (was 2258) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} accumulo-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} contrib in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} hbase-handler in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} hcatalog/core generated 0 new + 28 unchanged - 1
fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} util in the patch passed. {color} |
|
[jira] [Commented] (HIVE-21401) Break up DDLTask - extract Table related operations
[
https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790108#comment-16790108
]
Hive QA commented on HIVE-21401:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962017/HIVE-21401.12.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15825 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16454/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16454/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16454/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962017 - PreCommit-HIVE-Build
> Break up DDLTask - extract Table related operations
> ---
>
> Key: HIVE-21401
> URL: https://issues.apache.org/jira/browse/HIVE-21401
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch,
> HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch,
> HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch,
> HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch,
> HIVE-21401.12.patch
>
>
> DDLTask is a huge class, more than 5000 lines long. The related DDLWork is
> also a huge class, which has a field for each DDL operation it supports. The
> goal is to refactor these in order to have everything cut into more
> handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each operation
> * have a package for each operation group (database ddl, table ddl, etc), so
> the amount of classes under a package is more manageable
> * make all the requests (DDLDesc subclasses) immutable
> * DDLTask should be agnostic to the actual operations
> * right now let's ignore the issue of having some operations handled by
> DDLTask which are not actual DDL operations (lock, unlock, desc...)
> In the interim time when there are two DDLTask and DDLWork classes in the
> code base the new ones in the new package are called DDLTask2 and DDLWork2
> thus avoiding the usage of fully qualified class names where both the old and
> the new classes are in use.
> Step #2: extract all the table related operations from the old DDLTask except
> alter table, and move them under the new package. Also create the new
> internal framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
