[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723780#comment-16723780
]
Hive QA commented on HIVE-20911:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 12 new + 390 unchanged - 12
fixed = 402 total (was 402) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 714
unchanged - 10 fixed = 735 total (was 724) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
48s{color} | {color:red} ql generated 2 new + 2309 unchanged - 1 fixed = 2311
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field
org.apache.hadoop.hive.ql.exec.repl.ReplLoadWork.pathsToCopyIterator is
transient but isn't set by deserialization In ReplLoadWork.java:but isn't set
by deserialization In ReplLoadWork.java |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:[line 100] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15361/dev-support/hive-personality.sh
|
| git revision | master / 87f8ecc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15361/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723786#comment-16723786
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952144/HIVE-21001.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15362/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15362/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15362/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-12-18 07:52:29.480
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15362/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-12-18 07:52:29.484
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 87f8ecc HIVE-20785: Wrong key name in the JDBC
DatabaseMetaData.getPrimaryKeys method (Guillaume Grossetie reviewed by
Prasanth Jayachandran)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 87f8ecc HIVE-20785: Wrong key name in the JDBC
DatabaseMetaData.getPrimaryKeys method (Guillaume Grossetie reviewed by
Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-12-18 07:52:30.618
+ rm -rf ../yetus_PreCommit-HIVE-Build-15362
+ mkdir ../yetus_PreCommit-HIVE-Build-15362
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15362
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15362/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1840: trailing whitespace.
null sort order:
/data/hiveptest/working/scratch/build.patch:1841: trailing whitespace.
sort order:
/data/hiveptest/working/scratch/build.patch:1862: trailing whitespace.
columns.comments
/data/hiveptest/working/scratch/build.patch:1877: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:1887: trailing whitespace.
columns.comments
warning: squelched 27 whitespace errors
warning: 32 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.hive:hive:4.0.0-SNAPSHOT
(/data/hiveptest/working/apache-github-source-source/pom.xml) has 1 error
[ERROR] Malformed POM
/data/hiveptest/working/apache-github-source-source/pom.xml: Unrecognised tag:
'repositories' (position: START_TAG seen ...-- This needs to be removed before
checking in-->\n ... @224:17) @
/data/hiveptest/working/apache-github-source-source/pom.xml, line 224, column
17 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/ModelParseException
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15362
+ exit
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723785#comment-16723785
]
Hive QA commented on HIVE-20911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952129/HIVE-20911.05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15713 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testBootstrapLoadMigrationManagedToAcid
(batchId=243)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailure
(batchId=243)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailurePart
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15361/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15361/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15361/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952129 - PreCommit-HIVE-Build
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch, HIVE-20911.02.patch,
> HIVE-20911.03.patch, HIVE-20911.04.patch, HIVE-20911.05.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
[jira] [Updated] (HIVE-21054) union all query result empty
[
https://issues.apache.org/jira/browse/HIVE-21054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
daveKim updated HIVE-21054:
---
Description:
{code:java}
//
set hive.vectorized.execution.enabled=FALSE;
set hive.optimize.union.remove=FALSE;
SELECT distinct t1.ctype, t1.id_all , t2.list19
FROM (
SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
FROM tab WHERE stn1='20130101010100'
AND ctype22 BETWEEN 2 AND 5 --result 45 row
UNION ALL
SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
FROM space
WHERE stn1='20130102010100'
AND ctype22 BETWEEN 2 AND 5 --result empty
) t1 JOIN colon t2 ON (t1.id_all = t2.id20 AND t2.list19 ='ITEM30') -- join
matched 3 rows
{code}
expected result
||ctype||id_all||list19||
|2|104|ITEM30|
|2|683|ITEM30|
|1|970|ITEM30|
but, actual result empty..
instead "union all" below query result is not empty case are work.
{code:java}
// 코드 자리 표시자
set hive.vectorized.execution.enabled=FALSE;
set hive.optimize.union.remove=FALSE;
SELECT distinct t1.ctype, t1.id_all , t2.list19
FROM (
SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
FROM tab WHERE stn1='20130101010100'
AND ctype22 BETWEEN 2 AND 5 --result 45 row
UNION ALL
SELECT 1 AS ctype, '111' AS id_all, 'ITEM30' AS item_list --dummy row
) t1 JOIN colon t2 ON (t1.id_all = t2.id20 AND t2.list19 ='ITEM30') -- join
matched 3 rows
{code}
am i wrong properties or query?
reference
# HIVE-12788
# HIVE-20319
was:
{code:java}
//
set hive.vectorized.execution.enabled=FALSE;
set hive.optimize.union.remove=FALSE;
SELECT distinct t1.ctype, t1.id_all , t2.list19
FROM (
SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
FROM tab WHERE stn1='20130101010100'
AND ctype22 BETWEEN 2 AND 5 --result 45 row
UNION ALL
SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
FROM space
WHERE stn1='20130102010100'
AND ctype22 BETWEEN 2 AND 5 --result empty
) t1 JOIN colon t2 ON (t1.id_all = t2.id20 AND t2.list19 ='ITEM30') -- join
matched 3 rows
{code}
expected result
||ctype||id_all||list19||
|2|104|ITEM30|
|2|683|ITEM30|
|1|970|ITEM30|
but, actual result empty..
am i wrong properties or query?
reference
# HIVE-12788
# HIVE-20319
> union all query result empty
>
>
> Key: HIVE-21054
> URL: https://issues.apache.org/jira/browse/HIVE-21054
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 3.1.1
>Reporter: daveKim
>Priority: Major
>
>
>
> {code:java}
> //
> set hive.vectorized.execution.enabled=FALSE;
> set hive.optimize.union.remove=FALSE;
> SELECT distinct t1.ctype, t1.id_all , t2.list19
> FROM (
> SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
> FROM tab WHERE stn1='20130101010100'
> AND ctype22 BETWEEN 2 AND 5 --result 45 row
> UNION ALL
> SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
> FROM space
> WHERE stn1='20130102010100'
> AND ctype22 BETWEEN 2 AND 5 --result empty
> ) t1 JOIN colon t2 ON (t1.id_all = t2.id20 AND t2.list19 ='ITEM30') -- join
> matched 3 rows
> {code}
> expected result
> ||ctype||id_all||list19||
> |2|104|ITEM30|
> |2|683|ITEM30|
> |1|970|ITEM30|
> but, actual result empty..
>
> instead "union all" below query result is not empty case are work.
> {code:java}
> // 코드 자리 표시자
> set hive.vectorized.execution.enabled=FALSE;
> set hive.optimize.union.remove=FALSE;
> SELECT distinct t1.ctype, t1.id_all , t2.list19
> FROM (
> SELECT ctype22 AS ctype, id20 AS id_all, list19 AS item_list
> FROM tab WHERE stn1='20130101010100'
> AND ctype22 BETWEEN 2 AND 5 --result 45 row
> UNION ALL
> SELECT 1 AS ctype, '111' AS id_all, 'ITEM30' AS item_list --dummy row
> ) t1 JOIN colon t2 ON (t1.id_all = t2.id20 AND t2.list19 ='ITEM30') -- join
> matched 3 rows
> {code}
>
> am i wrong properties or query?
>
>
> reference
> # HIVE-12788
> # HIVE-20319
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.03.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723774#comment-16723774 ] Jesus Camacho Rodriguez commented on HIVE-21041: +1 (pending tests) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17718) Hive on Spark Debugging Improvements
[ https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned HIVE-17718: --- Assignee: (was: Chao Sun) > Hive on Spark Debugging Improvements > > > Key: HIVE-17718 > URL: https://issues.apache.org/jira/browse/HIVE-17718 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Priority: Major > > There are multiple places where it is hard to debug HoS - e.g. the HoS Remote > Driver and Client, the Spark RDD graph, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[
https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723756#comment-16723756
]
Hive QA commented on HIVE-21041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952128/HIVE-21041.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15730 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=273)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15360/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15360/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15360/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952128 - PreCommit-HIVE-Build
> NPE, ParseException in getting schema from logical plan
> ---
>
> Key: HIVE-21041
> URL: https://issues.apache.org/jira/browse/HIVE-21041
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch
>
>
> HIVE-20552 makes getting schema from logical plan faster. But it throws
> ParseException when it has column alias, and NullPointerException when it has
> subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723685#comment-16723685 ] ASF GitHub Bot commented on HIVE-21041: --- GitHub user pudidic opened a pull request: https://github.com/apache/hive/pull/507 HIVE-21041: NPE, ParseException in getting schema from logical plan (… …Teddy Choi) You can merge this pull request into a Git repository by running: $ git pull https://github.com/pudidic/hive HIVE-21041 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/507.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #507 commit 1b0a804e5ae77f17ccbfe91fb89849cba4ce79b5 Author: Teddy Choi Date: 2018-12-18T04:39:28Z HIVE-21041: NPE, ParseException in getting schema from logical plan (Teddy Choi) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-20141.3.patch, HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[
https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723716#comment-16723716
]
Hive QA commented on HIVE-21041:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15360/dev-support/hive-personality.sh
|
| git revision | master / 87f8ecc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15360/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE, ParseException in getting schema from logical plan
> ---
>
> Key: HIVE-21041
> URL: https://issues.apache.org/jira/browse/HIVE-21041
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch
>
>
> HIVE-20552 makes getting schema from logical plan faster. But it throws
> ParseException when it has column alias, and NullPointerException when it has
> subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723714#comment-16723714 ] Daniel Dai commented on HIVE-18661: --- +1 pending tests for 09.patch. > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch, HIVE-18661.09.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[
https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723702#comment-16723702
]
Hive QA commented on HIVE-21041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952119/HIVE-20552.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15730 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15359/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15359/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15359/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952119 - PreCommit-HIVE-Build
> NPE, ParseException in getting schema from logical plan
> ---
>
> Key: HIVE-21041
> URL: https://issues.apache.org/jira/browse/HIVE-21041
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch
>
>
> HIVE-20552 makes getting schema from logical plan faster. But it throws
> ParseException when it has column alias, and NullPointerException when it has
> subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: HIVE-20141.3.patch > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-20141.3.patch, HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20911:
---
Attachment: HIVE-20911.05.patch
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch, HIVE-20911.02.patch,
> HIVE-20911.03.patch, HIVE-20911.04.patch, HIVE-20911.05.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: HIVE-21041.3.patch > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: (was: HIVE-20141.3.patch) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-21041.2.patch, HIVE-21041.3.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[
https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723686#comment-16723686
]
Hive QA commented on HIVE-21041:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15359/dev-support/hive-personality.sh
|
| git revision | master / 87f8ecc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15359/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE, ParseException in getting schema from logical plan
> ---
>
> Key: HIVE-21041
> URL: https://issues.apache.org/jira/browse/HIVE-21041
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-20141.3.patch, HIVE-21041.2.patch
>
>
> HIVE-20552 makes getting schema from logical plan faster. But it throws
> ParseException when it has column alias, and NullPointerException when it has
> subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21041: -- Labels: pull-request-available (was: ) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-20141.3.patch, HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: (was: HIVE-20552.3.patch) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723650#comment-16723650 ] Teddy Choi commented on HIVE-21041: --- [~jcamachorodriguez], thanks for pointing out. In the third patch, I replaced analzyer.analyze() method call with analyzer.genLogicalPlan(). It works as we wanted. Here are the main causes I found; I got NPE for not having corresponding a Context object, and I got ParseException for not having a Context object with HIVE_QUOTEDID_SUPPORT configuration. So I made a context object with a given HiveConf object that has HIVE_QUOTEDID_SUPPORT configuration. I found no exception for these reasons. > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-20552.3.patch, HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: HIVE-20552.3.patch > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-20552.3.patch, HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: HIVE-20552.3.patch > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[ https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21041: -- Attachment: (was: HIVE-20552.3.patch) > NPE, ParseException in getting schema from logical plan > --- > > Key: HIVE-21041 > URL: https://issues.apache.org/jira/browse/HIVE-21041 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-21041.2.patch > > > HIVE-20552 makes getting schema from logical plan faster. But it throws > ParseException when it has column alias, and NullPointerException when it has > subqueries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21040) msck does unnecessary file listing at last level of directory tree
[
https://issues.apache.org/jira/browse/HIVE-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723637#comment-16723637
]
Hive QA commented on HIVE-21040:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952111/HIVE-21040.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15729 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15358/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15358/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15358/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952111 - PreCommit-HIVE-Build
> msck does unnecessary file listing at last level of directory tree
> --
>
> Key: HIVE-21040
> URL: https://issues.apache.org/jira/browse/HIVE-21040
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21040.01.patch
>
>
> Here is the code snippet which is run by {{msck}} to list directories
> {noformat}
> final Path currentPath = pd.p;
> final int currentDepth = pd.depth;
> FileStatus[] fileStatuses = fs.listStatus(currentPath,
> FileUtils.HIDDEN_FILES_PATH_FILTER);
> // found no files under a sub-directory under table base path; it is
> possible that the table
> // is empty and hence there are no partition sub-directories created
> under base path
> if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth <
> partColNames.size()) {
> // since maxDepth is not yet reached, we are missing partition
> // columns in currentPath
> logOrThrowExceptionWithMsg(
> "MSCK is missing partition columns under " +
> currentPath.toString());
> } else {
> // found files under currentPath add them to the queue if it is a
> directory
> for (FileStatus fileStatus : fileStatuses) {
> if (!fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a file at depth which is less than number of partition
> keys
> logOrThrowExceptionWithMsg(
> "MSCK finds a file rather than a directory when it searches
> for "
> + fileStatus.getPath().toString());
> } else if (fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a sub-directory at a depth less than number of partition
> keys
> // validate if the partition directory name matches with the
> corresponding
> // partition colName at currentDepth
> Path nextPath = fileStatus.getPath();
> String[] parts = nextPath.getName().split("=");
> if (parts.length != 2) {
> logOrThrowExceptionWithMsg("Invalid partition name " +
> nextPath);
> } else if
> (!parts[0].equalsIgnoreCase(partColNames.get(currentDepth))) {
> logOrThrowExceptionWithMsg(
> "Unexpected partition key " + parts[0] + " found at " +
> nextPath);
> } else {
> // add sub-directory to the work queue if maxDepth is not yet
> reached
> pendingPaths.add(new PathDepthInfo(nextPath, currentDepth + 1));
> }
> }
> }
> if (currentDepth == partColNames.size()) {
> return currentPath;
> }
> }
> {noformat}
> You can see that when the {{currentDepth}} at the {{maxDepth}} it still does
> a unnecessary listing of the files. We can improve this call by checking the
> currentDepth and bailing out early.
> This can improve the performance of msck command significantly especially
> when there are lot of files in each partitions on remote filesystems like S3
> or ADLS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21040) msck does unnecessary file listing at last level of directory tree
[
https://issues.apache.org/jira/browse/HIVE-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723612#comment-16723612
]
Hive QA commented on HIVE-21040:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
1s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15358/dev-support/hive-personality.sh
|
| git revision | master / 87f8ecc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15358/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> msck does unnecessary file listing at last level of directory tree
> --
>
> Key: HIVE-21040
> URL: https://issues.apache.org/jira/browse/HIVE-21040
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21040.01.patch
>
>
> Here is the code snippet which is run by {{msck}} to list directories
> {noformat}
> final Path currentPath = pd.p;
> final int currentDepth = pd.depth;
> FileStatus[] fileStatuses = fs.listStatus(currentPath,
> FileUtils.HIDDEN_FILES_PATH_FILTER);
> // found no files under a sub-directory under table base path; it is
> possible that the table
> // is empty and hence there are no partition sub-directories created
> under base path
> if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth <
> partColNames.size()) {
> // since maxDepth is not yet reached, we are missing partition
> // columns in currentPath
> logOrThrowExceptionWithMsg(
> "MSCK is missing partition columns under " +
> currentPath.toString());
> } else {
> // found files under currentPath add them to the queue if it is a
> directory
> for (FileStatus fileStatus : fileStatuses) {
> if (!fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a file at depth which is less than number of partition
> keys
>
[jira] [Updated] (HIVE-21040) msck does unnecessary file listing at last level of directory tree
[
https://issues.apache.org/jira/browse/HIVE-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21040:
---
Status: Patch Available (was: Open)
> msck does unnecessary file listing at last level of directory tree
> --
>
> Key: HIVE-21040
> URL: https://issues.apache.org/jira/browse/HIVE-21040
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21040.01.patch
>
>
> Here is the code snippet which is run by {{msck}} to list directories
> {noformat}
> final Path currentPath = pd.p;
> final int currentDepth = pd.depth;
> FileStatus[] fileStatuses = fs.listStatus(currentPath,
> FileUtils.HIDDEN_FILES_PATH_FILTER);
> // found no files under a sub-directory under table base path; it is
> possible that the table
> // is empty and hence there are no partition sub-directories created
> under base path
> if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth <
> partColNames.size()) {
> // since maxDepth is not yet reached, we are missing partition
> // columns in currentPath
> logOrThrowExceptionWithMsg(
> "MSCK is missing partition columns under " +
> currentPath.toString());
> } else {
> // found files under currentPath add them to the queue if it is a
> directory
> for (FileStatus fileStatus : fileStatuses) {
> if (!fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a file at depth which is less than number of partition
> keys
> logOrThrowExceptionWithMsg(
> "MSCK finds a file rather than a directory when it searches
> for "
> + fileStatus.getPath().toString());
> } else if (fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a sub-directory at a depth less than number of partition
> keys
> // validate if the partition directory name matches with the
> corresponding
> // partition colName at currentDepth
> Path nextPath = fileStatus.getPath();
> String[] parts = nextPath.getName().split("=");
> if (parts.length != 2) {
> logOrThrowExceptionWithMsg("Invalid partition name " +
> nextPath);
> } else if
> (!parts[0].equalsIgnoreCase(partColNames.get(currentDepth))) {
> logOrThrowExceptionWithMsg(
> "Unexpected partition key " + parts[0] + " found at " +
> nextPath);
> } else {
> // add sub-directory to the work queue if maxDepth is not yet
> reached
> pendingPaths.add(new PathDepthInfo(nextPath, currentDepth + 1));
> }
> }
> }
> if (currentDepth == partColNames.size()) {
> return currentPath;
> }
> }
> {noformat}
> You can see that when the {{currentDepth}} at the {{maxDepth}} it still does
> a unnecessary listing of the files. We can improve this call by checking the
> currentDepth and bailing out early.
> This can improve the performance of msck command significantly especially
> when there are lot of files in each partitions on remote filesystems like S3
> or ADLS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21040) msck does unnecessary file listing at last level of directory tree
[
https://issues.apache.org/jira/browse/HIVE-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723595#comment-16723595
]
Vihang Karajgaonkar commented on HIVE-21040:
I spent a lot of time to figure out a good way to test this. {{FileSystem}}
actually provides APIs to get statistics but for some reason I am not able to
use it in the test framework to confirm that number listStatus calls are as
expected. Will try to dig more into it. If anyone has more ideas, please let me
know.
> msck does unnecessary file listing at last level of directory tree
> --
>
> Key: HIVE-21040
> URL: https://issues.apache.org/jira/browse/HIVE-21040
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21040.01.patch
>
>
> Here is the code snippet which is run by {{msck}} to list directories
> {noformat}
> final Path currentPath = pd.p;
> final int currentDepth = pd.depth;
> FileStatus[] fileStatuses = fs.listStatus(currentPath,
> FileUtils.HIDDEN_FILES_PATH_FILTER);
> // found no files under a sub-directory under table base path; it is
> possible that the table
> // is empty and hence there are no partition sub-directories created
> under base path
> if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth <
> partColNames.size()) {
> // since maxDepth is not yet reached, we are missing partition
> // columns in currentPath
> logOrThrowExceptionWithMsg(
> "MSCK is missing partition columns under " +
> currentPath.toString());
> } else {
> // found files under currentPath add them to the queue if it is a
> directory
> for (FileStatus fileStatus : fileStatuses) {
> if (!fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a file at depth which is less than number of partition
> keys
> logOrThrowExceptionWithMsg(
> "MSCK finds a file rather than a directory when it searches
> for "
> + fileStatus.getPath().toString());
> } else if (fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a sub-directory at a depth less than number of partition
> keys
> // validate if the partition directory name matches with the
> corresponding
> // partition colName at currentDepth
> Path nextPath = fileStatus.getPath();
> String[] parts = nextPath.getName().split("=");
> if (parts.length != 2) {
> logOrThrowExceptionWithMsg("Invalid partition name " +
> nextPath);
> } else if
> (!parts[0].equalsIgnoreCase(partColNames.get(currentDepth))) {
> logOrThrowExceptionWithMsg(
> "Unexpected partition key " + parts[0] + " found at " +
> nextPath);
> } else {
> // add sub-directory to the work queue if maxDepth is not yet
> reached
> pendingPaths.add(new PathDepthInfo(nextPath, currentDepth + 1));
> }
> }
> }
> if (currentDepth == partColNames.size()) {
> return currentPath;
> }
> }
> {noformat}
> You can see that when the {{currentDepth}} at the {{maxDepth}} it still does
> a unnecessary listing of the files. We can improve this call by checking the
> currentDepth and bailing out early.
> This can improve the performance of msck command significantly especially
> when there are lot of files in each partitions on remote filesystems like S3
> or ADLS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21040) msck does unnecessary file listing at last level of directory tree
[
https://issues.apache.org/jira/browse/HIVE-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-21040:
---
Attachment: HIVE-21040.01.patch
> msck does unnecessary file listing at last level of directory tree
> --
>
> Key: HIVE-21040
> URL: https://issues.apache.org/jira/browse/HIVE-21040
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21040.01.patch
>
>
> Here is the code snippet which is run by {{msck}} to list directories
> {noformat}
> final Path currentPath = pd.p;
> final int currentDepth = pd.depth;
> FileStatus[] fileStatuses = fs.listStatus(currentPath,
> FileUtils.HIDDEN_FILES_PATH_FILTER);
> // found no files under a sub-directory under table base path; it is
> possible that the table
> // is empty and hence there are no partition sub-directories created
> under base path
> if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth <
> partColNames.size()) {
> // since maxDepth is not yet reached, we are missing partition
> // columns in currentPath
> logOrThrowExceptionWithMsg(
> "MSCK is missing partition columns under " +
> currentPath.toString());
> } else {
> // found files under currentPath add them to the queue if it is a
> directory
> for (FileStatus fileStatus : fileStatuses) {
> if (!fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a file at depth which is less than number of partition
> keys
> logOrThrowExceptionWithMsg(
> "MSCK finds a file rather than a directory when it searches
> for "
> + fileStatus.getPath().toString());
> } else if (fileStatus.isDirectory() && currentDepth <
> partColNames.size()) {
> // found a sub-directory at a depth less than number of partition
> keys
> // validate if the partition directory name matches with the
> corresponding
> // partition colName at currentDepth
> Path nextPath = fileStatus.getPath();
> String[] parts = nextPath.getName().split("=");
> if (parts.length != 2) {
> logOrThrowExceptionWithMsg("Invalid partition name " +
> nextPath);
> } else if
> (!parts[0].equalsIgnoreCase(partColNames.get(currentDepth))) {
> logOrThrowExceptionWithMsg(
> "Unexpected partition key " + parts[0] + " found at " +
> nextPath);
> } else {
> // add sub-directory to the work queue if maxDepth is not yet
> reached
> pendingPaths.add(new PathDepthInfo(nextPath, currentDepth + 1));
> }
> }
> }
> if (currentDepth == partColNames.size()) {
> return currentPath;
> }
> }
> {noformat}
> You can see that when the {{currentDepth}} at the {{maxDepth}} it still does
> a unnecessary listing of the files. We can improve this call by checking the
> currentDepth and bailing out early.
> This can improve the performance of msck command significantly especially
> when there are lot of files in each partitions on remote filesystems like S3
> or ADLS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723564#comment-16723564 ] Eugene Koifman commented on HIVE-20936: --- added some RB comments > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.10.patch, > HIVE-20936.11.patch, HIVE-20936.2.patch, HIVE-20936.3.patch, > HIVE-20936.4.patch, HIVE-20936.5.patch, HIVE-20936.6.patch, > HIVE-20936.7.patch, HIVE-20936.8.patch, HIVE-20936.8.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723535#comment-16723535
]
Hive QA commented on HIVE-20785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952095/HIVE-20785.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15728 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15357/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15357/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15357/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952095 - PreCommit-HIVE-Build
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20785:
-
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Committed to branch-3 and master. Thanks [~ggrossetie] for the contribution!
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19081:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Release Note: n/a
Status: Resolved (was: Patch Available)
committed to master
thanks Igor for the contribution
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723507#comment-16723507
]
Hive QA commented on HIVE-20785:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} service: The patch generated 1 new + 6 unchanged - 1
fixed = 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15357/dev-support/hive-personality.sh
|
| git revision | master / 4d9df0f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15357/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15357/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20785:
-
Attachment: HIVE-20785.1.patch
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723465#comment-16723465
]
Prasanth Jayachandran commented on HIVE-20785:
--
{code:java}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.common.metrics.metrics2.TestCodahaleMetrics.testFileReporting
(batchId=282){code}
These 2 test failures seems to be unrelated. Giving the patch another try to
see if the tests are flaky.
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-20785:
Assignee: Prasanth Jayachandran (was: Guillaume Grossetie)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-20785:
Assignee: Guillaume Grossetie (was: Prasanth Jayachandran)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: HIVE-20785.1.patch, patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21052) Make sure transaction get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M reassigned HIVE-21052: -- Assignee: Jaume M > Make sure transaction get cleaned if they are aborted before addPartitions is > called > > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723424#comment-16723424
]
Hive QA commented on HIVE-19081:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952078/HIVE-19081.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 15725 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers1
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testMultipleTriggers2
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomReadOps
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagRawInputSplitsKill
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagTotalTasks
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDefaultRawInputSplits
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesWrite
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerShortQueryElapsedTime
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerTotalTasks
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsKill
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerVertexRawInputSplitsNoKill
(batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15355/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15355/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15355/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952078 - PreCommit-HIVE-Build
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723415#comment-16723415 ] Vihang Karajgaonkar commented on HIVE-20992: Patch merged into master. Thanks for your contribution [~mramdenbourg] > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, > HIVE-20992.7.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20992: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, > HIVE-20992.7.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-951) Selectively include EXTERNAL TABLE source files via REGEX
[
https://issues.apache.org/jira/browse/HIVE-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723408#comment-16723408
]
Amit Banerjee commented on HIVE-951:
This doesn't seem to work anymore. Was this feature discontinued somewhere as I
have the same requirement.
> Selectively include EXTERNAL TABLE source files via REGEX
> -
>
> Key: HIVE-951
> URL: https://issues.apache.org/jira/browse/HIVE-951
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Carl Steinbach
>Priority: Major
> Attachments: HIVE-951.patch
>
>
> CREATE EXTERNAL TABLE should allow users to cherry-pick files via regular
> expression.
> CREATE EXTERNAL TABLE was designed to allow users to access data that exists
> outside of Hive, and
> currently makes the assumption that all of the files located under the
> supplied path should be included
> in the new table. Users frequently encounter directories containing multiple
> datasets, or directories that contain data in heterogeneous schemas, and it's
> often
> impractical or impossible to adjust the layout of the directory to meet the
> requirements of
> CREATE EXTERNAL TABLE. A good example of this problem is creating an external
> table based
> on the contents of an S3 bucket.
> One way to solve this problem is to extend the syntax of CREATE EXTERNAL TABLE
> as follows:
> CREATE EXTERNAL TABLE
> ...
> LOCATION path [file_regex]
> ...
> For example:
> {code:sql}
> CREATE EXTERNAL TABLE mytable1 ( a string, b string, c string )
> STORED AS TEXTFILE
> LOCATION 's3://my.bucket/' 'folder/2009.*\.bz2$';
> {code}
> Creates mytable1 which includes all files in s3:/my.bucket with a filename
> matching 'folder/2009*.bz2'
> {code:sql}
> CREATE EXTERNAL TABLE mytable2 ( d string, e int, f int, g int )
> STORED AS TEXTFILE
> LOCATION 'hdfs://data/' 'xyz.*2009.bz2$';
> {code}
> Creates mytable2 including all files matching 'xyz*2009.bz2' located
> under hdfs://data/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723398#comment-16723398
]
Eugene Koifman commented on HIVE-19081:
---
+1 patch 2
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723384#comment-16723384
]
Hive QA commented on HIVE-19081:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 386 unchanged - 2
fixed = 387 total (was 388) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15355/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15355/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15355/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-20992: - Attachment: HIVE-20992.7.patch Status: Patch Available (was: In Progress) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, > HIVE-20992.7.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-20992: - Status: In Progress (was: Patch Available) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, > HIVE-20992.7.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20959) cbo_rp_limit / cbo_limit are flaky - intermittent whitespace difference
[
https://issues.apache.org/jira/browse/HIVE-20959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723350#comment-16723350
]
Vihang Karajgaonkar commented on HIVE-20959:
Yeah, I tried running it locally and running with the same batch as well. Not
sure why it fails intermittently. If you are seeing tab instead of space, can
you force the test to use one of the two so that it is consistent and not
environment dependent?
> cbo_rp_limit / cbo_limit are flaky - intermittent whitespace difference
> ---
>
> Key: HIVE-20959
> URL: https://issues.apache.org/jira/browse/HIVE-20959
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments:
> 171-TestMiniLlapLocalCliDriver-dynamic_semijoin_reduction.q-materialized_view_create_rewrite_3.q-vectorization_pushdown.q-and-27-more.txt,
>
> TEST-171-TestMiniLlapLocalCliDriver-dynamic_semijoin_reduction.q-materialized_view_create_rewrite_3.q-vectorization_pushdown.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.xml,
> diff
>
>
> {code:java}
> Client Execution succeeded but contained differences (error code = 1) after
> executing cbo_rp_limit.q
> 11c11
> < 1 4 2
> ---
> > 1 4 2
> {code}
> After copying here to jira, cannot found difference, but by checking from
> original junit xml, there is a whitespace difference in the lines
> (represented as hex values), between 1 (x31) and 4 (x34). See [^diff] .
> Original golden file contains horizontal tab (x09), actual output contains
> space (x20).
> The serious thing is that the separator changes to x20, which is wrong, but
> then in the same line, it changes back to x09.
> {code}
> 20 31 *20* 34 09 32 <- actual
> 20 31 *09* 34 09 32 <- expected
> {code}
> Tried to reproduce it by running the failing batch of qtests locally, but no
> luck (maybe it's an environment issue)
> {code}
> mvn test -T 1C -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl
> itests/qtest -pl itests/util -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=dynamic_semijoin_reduction.q,materialized_view_create_rewrite_3.q,vectorization_pushdown.q,correlationoptimizer2.q,cbo_gby_empty.q,schema_evol_text_nonvec_part_all_complex_llap_io.q,vectorization_short_regress.q,mapjoin3.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,unionDistinct_3.q,cbo_join.q,correlationoptimizer6.q,union_remove_26.q,cbo_rp_limit.q,convert_decimal64_to_decimal.q,vector_groupby_cube1.q,union2.q,groupby2.q,dynpart_sort_opt_vectorization.q,constraints_optimization.q,exchgpartition2lel.q,retry_failure.q,schema_evol_text_vecrow_part_llap_io.q,sample10.q,vectorized_timestamp_ints_casts.q,auto_sortmerge_join_2.q,bucketizedhiveinputformat.q,cte_mat_2.q,vectorization_8.q
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723349#comment-16723349 ] Vihang Karajgaonkar commented on HIVE-20992: Hi [~mramdenbourg] Left couple of minor edits to the patch on the review board. Rest looks good. I don't think we need to wait for full tests given most of the changes suggested are trivial. Just make sure that you run the newly added test locally. +1 > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-19081:
-
Attachment: HIVE-19081.02.patch
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723322#comment-16723322
]
Igor Kryvenko commented on HIVE-19081:
--
Failed tests are not related. Attached patch #2 to verify it.
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch, HIVE-19081.02.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21041) NPE, ParseException in getting schema from logical plan
[
https://issues.apache.org/jira/browse/HIVE-21041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723304#comment-16723304
]
Jesus Camacho Rodriguez commented on HIVE-21041:
[~teddy.choi], I have quickly checked the patch and it seems to me that it
defeats the original purpose of HIVE-20552, since the {{analyze}} method will
go through all the plan generation steps (including physical optimizer, Tez
compilation, etc.).
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L12255
Have you checked why we are getting those errors? Maybe it is a problem with
the planner? Are we always going through Calcite, e.g., is it because of any
feature that is being used and it is not supported by CBO? If that is the case,
I believe a better option would be to try to get the schema through the
{{genLogicalPlan}} method, and if the plan obtained it is null, fall back and
run the {{analyze}} method. At least this will give you the fast path for most
of the queries. In any case, it is important to understand why you have not
been able to retrieve the schema in all cases.
> NPE, ParseException in getting schema from logical plan
> ---
>
> Key: HIVE-21041
> URL: https://issues.apache.org/jira/browse/HIVE-21041
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-21041.2.patch
>
>
> HIVE-20552 makes getting schema from logical plan faster. But it throws
> ParseException when it has column alias, and NullPointerException when it has
> subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723306#comment-16723306
]
Hive QA commented on HIVE-19081:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952064/HIVE-19081.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15725 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15354/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15354/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15354/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952064 - PreCommit-HIVE-Build
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723273#comment-16723273
]
Hive QA commented on HIVE-19081:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 2 new + 386 unchanged - 2
fixed = 388 total (was 388) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15354/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15354/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15354/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723260#comment-16723260
]
Hive QA commented on HIVE-21030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952057/HIVE-21030.branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 106 failed/errored test(s), 14474 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=274)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=274)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=239)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestGetPartitionsUsingProjectionAndFilterSpecs - did not produce a TEST-*.xml
file (likely timed out) (batchId=230)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=236)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=236)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=233)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=236)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestMarkPartitionRemote - did not produce a
[jira] [Updated] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-19081:
-
Attachment: HIVE-19081.01.patch
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723227#comment-16723227 ] Jaume M commented on HIVE-20936: Can you rereview [~ekoifman] > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.10.patch, > HIVE-20936.11.patch, HIVE-20936.2.patch, HIVE-20936.3.patch, > HIVE-20936.4.patch, HIVE-20936.5.patch, HIVE-20936.6.patch, > HIVE-20936.7.patch, HIVE-20936.8.patch, HIVE-20936.8.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-19081:
-
Status: Patch Available (was: In Progress)
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19081.01.patch
>
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[
https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723175#comment-16723175
]
Hive QA commented on HIVE-21030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-15353/patches/PreCommit-HIVE-Build-15353.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15353/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add credential store env properties redaction in JobConf
>
>
> Key: HIVE-21030
> URL: https://issues.apache.org/jira/browse/HIVE-21030
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch,
> HIVE-21030.3.patch, HIVE-21030.4.patch, HIVE-21030.5.patch,
> HIVE-21030.6.patch, HIVE-21030.branch-3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723148#comment-16723148 ] Denys Kuzmenko commented on HIVE-21030: --- [~vihangk1] Thank you for the review! I've added patch for branch-3 as well. > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch, > HIVE-21030.3.patch, HIVE-21030.4.patch, HIVE-21030.5.patch, > HIVE-21030.6.patch, HIVE-21030.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21030) Add credential store env properties redaction in JobConf
[ https://issues.apache.org/jira/browse/HIVE-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21030: -- Attachment: HIVE-21030.branch-3.patch > Add credential store env properties redaction in JobConf > > > Key: HIVE-21030 > URL: https://issues.apache.org/jira/browse/HIVE-21030 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21030.1.patch, HIVE-21030.2.patch, > HIVE-21030.3.patch, HIVE-21030.4.patch, HIVE-21030.5.patch, > HIVE-21030.6.patch, HIVE-21030.branch-3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned HIVE-20776: Assignee: Na Li > Run HMS filterHooks on server-side in addition to client-side > - > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Assignee: Na Li >Priority: Major > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-21051) HMS server should authorize metadata read operation and and filter its result
[ https://issues.apache.org/jira/browse/HIVE-21051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li resolved HIVE-21051. -- Resolution: Duplicate duplicate of https://issues.apache.org/jira/browse/HIVE-20776 > HMS server should authorize metadata read operation and and filter its result > - > > Key: HIVE-21051 > URL: https://issues.apache.org/jira/browse/HIVE-21051 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > > Move all filter hooks from HMS client to HMS server (this will take care of > the filters for get_table/database/index, etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Manamcheri reassigned HIVE-20776: - Assignee: (was: Karthik Manamcheri) > Run HMS filterHooks on server-side in addition to client-side > - > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Priority: Major > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21051) HMS server should authorize metadata read operation and and filter its result
[ https://issues.apache.org/jira/browse/HIVE-21051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned HIVE-21051: > HMS server should authorize metadata read operation and and filter its result > - > > Key: HIVE-21051 > URL: https://issues.apache.org/jira/browse/HIVE-21051 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > > Move all filter hooks from HMS client to HMS server (this will take care of > the filters for get_table/database/index, etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21050) Upgrade Parquet to 1.12.0 and use LogicalTypes
[ https://issues.apache.org/jira/browse/HIVE-21050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-21050: > Upgrade Parquet to 1.12.0 and use LogicalTypes > -- > > Key: HIVE-21050 > URL: https://issues.apache.org/jira/browse/HIVE-21050 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: Parquet, parquet > > [WIP; contains necessary jars until Parquet community releases version 1.12.0] > The new Parquet version (1.12.0) uses > [LogicalTypes|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md] > instead of OriginalTypes. > These are backwards-compatible with OriginalTypes. > Thanks to [~kuczoram] for her work on this patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722944#comment-16722944
]
Hive QA commented on HIVE-20785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952021/patch.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15352/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15352/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15352/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12952021/patch.patch was found
in seen patch url's cache and a test was probably run already on it. Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952021 - PreCommit-HIVE-Build
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722942#comment-16722942
]
Hive QA commented on HIVE-20785:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952021/patch.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15724 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.common.metrics.metrics2.TestCodahaleMetrics.testFileReporting
(batchId=282)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15351/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15351/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15351/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952021 - PreCommit-HIVE-Build
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722901#comment-16722901
]
Hive QA commented on HIVE-20785:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} service: The patch generated 1 new + 6 unchanged - 1
fixed = 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15351/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15351/yetus/diff-checkstyle-service.txt
|
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15351/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722892#comment-16722892
]
Hive QA commented on HIVE-20911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952004/HIVE-20911.04.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=85)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testBootstrapLoadMigrationManagedToAcid
(batchId=243)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailure
(batchId=243)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.testIncrementalLoadMigrationManagedToAcidFailurePart
(batchId=243)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=258)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15350/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15350/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15350/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952004 - PreCommit-HIVE-Build
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch, HIVE-20911.02.patch,
> HIVE-20911.03.patch, HIVE-20911.04.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: patch.patch
Status: Patch Available (was: Open)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722885#comment-16722885
]
Hive QA commented on HIVE-20911:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 12 new + 390 unchanged - 12
fixed = 402 total (was 402) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 714
unchanged - 10 fixed = 735 total (was 724) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 2 new + 2309 unchanged - 1 fixed = 2311
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field
org.apache.hadoop.hive.ql.exec.repl.ReplLoadWork.pathsToCopyIterator is
transient but isn't set by deserialization In ReplLoadWork.java:but isn't set
by deserialization In ReplLoadWork.java |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger, ReplLoadWork, TaskTracker) At
IncrementalLoadTasksBuilder.java:[line 100] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15350/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15350/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: (was: patch.patch)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Status: Open (was: Patch Available)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21048) Remove needless org.mortbay.jetty from hadoop exclusions
[
https://issues.apache.org/jira/browse/HIVE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722836#comment-16722836
]
Hive QA commented on HIVE-21048:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
11s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15349/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| modules | C: storage-api common llap-tez ql service jdbc hcatalog
hcatalog/core hcatalog/hcatalog-pig-adapter hcatalog/webhcat/svr .
itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15349/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove needless org.mortbay.jetty from hadoop exclusions
>
>
> Key: HIVE-21048
> URL: https://issues.apache.org/jira/browse/HIVE-21048
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21048.01.patch, HIVE-21048.02.patch,
> HIVE-21048.03.patch, dep.out
>
>
> During HIVE-20638 i found that org.mortbay.jetty exclusions from e.g. hadoop
> don't take effect, as the actual groupId of jetty is org.eclipse.jetty for
> most of the current projects, please find attachment (example for hive
> commons project).
> https://en.wikipedia.org/wiki/Jetty_(web_server)#History
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21048) Remove needless org.mortbay.jetty from hadoop exclusions
[
https://issues.apache.org/jira/browse/HIVE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722839#comment-16722839
]
Hive QA commented on HIVE-21048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951999/HIVE-21048.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15724 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=272)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=272)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_csv]
(batchId=272)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=272)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=261)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15349/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15349/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15349/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951999 - PreCommit-HIVE-Build
> Remove needless org.mortbay.jetty from hadoop exclusions
>
>
> Key: HIVE-21048
> URL: https://issues.apache.org/jira/browse/HIVE-21048
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21048.01.patch, HIVE-21048.02.patch,
> HIVE-21048.03.patch, dep.out
>
>
> During HIVE-20638 i found that org.mortbay.jetty exclusions from e.g. hadoop
> don't take effect, as the actual groupId of jetty is org.eclipse.jetty for
> most of the current projects, please find attachment (example for hive
> commons project).
> https://en.wikipedia.org/wiki/Jetty_(web_server)#History
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: patch.patch
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: (was: patch.patch)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: (was: 20785.patch)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20785) Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
[
https://issues.apache.org/jira/browse/HIVE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume Grossetie updated HIVE-20785:
---
Attachment: (was: 20785.patch)
> Wrong key name in the JDBC DatabaseMetaData.getPrimaryKeys method
> -
>
> Key: HIVE-20785
> URL: https://issues.apache.org/jira/browse/HIVE-20785
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.0
>Reporter: Guillaume Grossetie
>Assignee: Guillaume Grossetie
>Priority: Major
> Attachments: patch.patch
>
>
> According to the documentation (1) the key should be {{KEY_SEQ, not KEQ_SEQ.}}
> Pull request available: https://github.com/apache/hive/pull/440
>
> (1)
> [https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getPrimaryKeys-java.lang.String-java.lang.String-java.lang.String-]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20911) External Table Replication for Hive
[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek updated HIVE-20911:
---
Attachment: HIVE-20911.04.patch
> External Table Replication for Hive
> ---
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20911.01.patch, HIVE-20911.02.patch,
> HIVE-20911.03.patch, HIVE-20911.04.patch
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> tableName,base64Encoded(tableDataLocation)
> {code}
> In case there are different partitions in the table pointing to different
> locations there will be multiple entries in the file for the same table name
> with location pointing to different partition locations. For partitions
> created in a table without specifying the _set location_ command will be
> within the same table Data location and hence there will not be different
> entries in the file above
> ** *repl load* will read the *\_external\_tables\_info* to identify what
> locations are to be copied from source to target and create corresponding
> tasks for them.
> * New External tables will be created with metadata only with no data copied
> as part of regular tasks while incremental load/bootstrap load.
> * Bootstrap dump will also create *\_external\_tables\_info* which will be
> used to copy data from source to target as part of boostrap load.
> * Bootstrap load will create a DAG, that can use parallelism in the execution
> phase, the hdfs copy related tasks are created, once the bootstrap phase is
> complete.
> * Since incremental load results in a DAG with only sequential execution (
> events applied in sequence ) to effectively use the parallelism capability in
> execution mode, we create tasks for hdfs copy along with the incremental DAG.
> This requires a few basic calculations to approximately meet the configured
> value in "hive.repl.approx.max.load.tasks"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21048) Remove needless org.mortbay.jetty from hadoop exclusions
[ https://issues.apache.org/jira/browse/HIVE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21048: Attachment: HIVE-21048.03.patch > Remove needless org.mortbay.jetty from hadoop exclusions > > > Key: HIVE-21048 > URL: https://issues.apache.org/jira/browse/HIVE-21048 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21048.01.patch, HIVE-21048.02.patch, > HIVE-21048.03.patch, dep.out > > > During HIVE-20638 i found that org.mortbay.jetty exclusions from e.g. hadoop > don't take effect, as the actual groupId of jetty is org.eclipse.jetty for > most of the current projects, please find attachment (example for hive > commons project). > https://en.wikipedia.org/wiki/Jetty_(web_server)#History -- This message was sent by Atlassian JIRA (v7.6.3#76005)
