[jira] [Commented] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802507#comment-16802507
]
Hive QA commented on HIVE-21230:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963809/HIVE-21230.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15846 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16705/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16705/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16705/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963809 - PreCommit-HIVE-Build
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802509#comment-16802509
]
Hive QA commented on HIVE-14719:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12827869/HIVE-14719.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16706/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16706/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16706/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-27 07:29:43.506
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16706/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-27 07:29:43.525
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5708a0b HIVE-21479: NPE during metastore cache update (Daniel
Dai, reviewed by Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5708a0b HIVE-21479: NPE during metastore cache update (Daniel
Dai, reviewed by Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-27 07:29:44.944
+ rm -rf ../yetus_PreCommit-HIVE-Build-16706
+ mkdir ../yetus_PreCommit-HIVE-Build-16706
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16706
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16706/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java: does not
exist in index
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java:22
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java:22
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16706
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12827869 - PreCommit-HIVE-Build
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802511#comment-16802511
]
Vaibhav Gumashta commented on HIVE-20580:
-
+1
[~pvary] Your patch looks good. I think with regards to {{isOriginal(Footer)}},
you might have hit a separate bug - both these methods should return the same
value.
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802532#comment-16802532
]
Hive QA commented on HIVE-21231:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 11
fixed = 16 total (was 27) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16707/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16707/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16707/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802545#comment-16802545
]
Peter Vary commented on HIVE-21402:
---
[~vgumashta]: The rethrown throwable will be caught again a few lines below,
and will be printed again here:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java#L239]
Basically we finish the worker job anyway (with or without the patch), just the
status for the compaction is not updated to match this before the patch. I
think this is the only change the fix does.
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219202 ] ASF GitHub Bot logged work on HIVE-21109: - Author: ASF GitHub Bot Created on: 27/Mar/19 08:41 Start Date: 27/Mar/19 08:41 Worklog Time Spent: 10m Work Description: ashutosh-bapat commented on pull request #579: HIVE-21109 : Support stats replication for ACID tables. URL: https://github.com/apache/hive/pull/579#discussion_r269452642 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java ## @@ -199,12 +199,15 @@ private AddPartitionDesc partitionDesc(Path fromPath, // Right now, we do not have a way of associating a writeId with statistics for a table // converted to a transactional table if it was non-transactional on the source. So, do not Review comment: Done. Looks like I missed pushing an entire commit fixing the comments. Done now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 219202) Time Spent: 3h 10m (was: 3h) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802555#comment-16802555 ] Zoltan Haindrich commented on HIVE-21516: - * please use unlink instead of rm -rf of a single file * could you chdir into the HIVE_ROOT at the start of the script instead of working with relative paths (../../ crap) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21521) Upgrade Hive to use ORC 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned HIVE-21521: -- > Upgrade Hive to use ORC 1.5.5 > - > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802556#comment-16802556
]
Hive QA commented on HIVE-21231:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963832/HIVE-21231.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15846 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_by_range_rule_not_null]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16707/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16707/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16707/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963832 - PreCommit-HIVE-Build
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Summary: Upgrade ORC to 1.5.5 (was: Upgrade Hive to use ORC 1.5.5) > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-21455:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for [~mszurap] for the patch and [~belugabehr] for the review!
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-21455:
-
Assignee: Miklos Szurap (was: S. Poortman19-88)
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.0, 1.1.0, 2.0.0, 2.1.0, 3.0.0, 3.1.0
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.2.patch, HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Attachment: HIVE-21521.01.patch Status: Patch Available (was: Open) > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Description: ORC-476 Make SearchAgument kryo buffer size configurable. > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC-476 Make SearchAgument kryo buffer size configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Status: Patch Available (was: Open)
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Status: Open (was: Patch Available)
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Attachment: HIVE-21511.2.patch
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Description: ORC-476 Make SearchAgument kryo buffer size configurable. Release notes [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] was:ORC-476 Make SearchAgument kryo buffer size configurable. > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC-476 Make SearchAgument kryo buffer size configurable. > > Release notes > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Description: ORC 1.5.5 release notes: [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo buffer overflow. I'd like we can back port this patch to branch-2.x was: ORC-476 Make SearchAgument kryo buffer size configurable. Release notes [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow. I'd like we can back port this patch to branch-2.x > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Description: ORC 1.5.5 release notes: [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo buffer overflow(more details). was: ORC 1.5.5 release notes: [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo buffer overflow. I'd like we can back port this patch to branch-2.x > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow(more details). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802564#comment-16802564 ] Yuming Wang commented on HIVE-21521: I'd like we can back port this patch to branch-2.x. > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow(more details). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802565#comment-16802565 ] Yuming Wang commented on HIVE-21521: cc [~vgumashta] > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow(more details). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated HIVE-21521: --- Description: ORC 1.5.5 release notes: [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo buffer overflow([more details|https://issues.apache.org/jira/browse/SPARK-27107]). was: ORC 1.5.5 release notes: [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo buffer overflow(more details). > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow([more > details|https://issues.apache.org/jira/browse/SPARK-27107]). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802569#comment-16802569
]
Hive QA commented on HIVE-21516:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16708/dev-support/hive-personality.sh
|
| git revision | master / b389484 |
| Default Java | 1.8.0_111 |
| modules | C: itests itests/hive-unit itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16708/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix spark downloading for q tests
> -
>
> Key: HIVE-21516
> URL: https://issues.apache.org/jira/browse/HIVE-21516
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch
>
>
> Currently itests/pom.xml declares a command to generated the download script
> for spark, thus it is re-generated every time any maven command is executed
> for any sub project of itests. AS a side effect it is leaving download.sh
> files everywhere. The download.sh file is almost totally static, no need to
> recreate it every time, just requires $spark.version as a parameter.
> Also it is only working properly under linux, as it relies on the md5sum
> program which is not present in OS X. This means that if the spark tarball is
> partially downloaded on OS X, then it would never be re-downloaded. This
> should be fixed by making it work as well using md5 on OS X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219208&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219208
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:14
Start Date: 27/Mar/19 09:14
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269463941
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -1894,6 +1898,16 @@ private void create_table_core(final RawStore ms, final
Table tbl,
List checkConstraints)
throws AlreadyExistsException, MetaException,
InvalidObjectException, NoSuchObjectException, InvalidInputException {
+
+ ColumnStatistics colStats = null;
+ // If the given table has column statistics, save it here. We will
update it later.
+ // We don't want it to be part of the Table object being created, lest
the create table
Review comment:
" and also shouldn't be persisted". That's not true. We will persist the
table stats but later. If you let me know which part of the comment is complex
(needs simplification), will come up with alternate wording reflecting the same
meaning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219208)
Time Spent: 3h 20m (was: 3h 10m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797986#comment-16797986
]
Denys Kuzmenko edited comment on HIVE-21397 at 3/27/19 9:17 AM:
Attached patch
[HIVE-21397.patch|https://issues.apache.org/jira/secure/attachment/12963255/HIVE-21397.patch]
Should be submitted once
[ORC-477|https://issues.apache.org/jira/browse/ORC-477] is merged into
branch-1.5 and orc-core v1.5.6 is released.
was (Author: dkuzmenko):
Attached patch
[HIVE-21397.patch|https://issues.apache.org/jira/secure/attachment/12963255/HIVE-21397.patch]
Should be submitted once
[ORC-477|https://issues.apache.org/jira/browse/ORC-477] is merged into
branch-1.5 and orc-core v1.5.5 is released.
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is
[jira] [Comment Edited] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797986#comment-16797986
]
Denys Kuzmenko edited comment on HIVE-21397 at 3/27/19 9:18 AM:
Attached patch
[HIVE-21397.patch|https://issues.apache.org/jira/secure/attachment/12963255/HIVE-21397.patch]
Should be submitted once
[ORC-477|https://issues.apache.org/jira/browse/ORC-477] is merged into
branch-1.5 and orc-core v1.5.6 is released (sometime in mid April).
was (Author: dkuzmenko):
Attached patch
[HIVE-21397.patch|https://issues.apache.org/jira/secure/attachment/12963255/HIVE-21397.patch]
Should be submitted once
[ORC-477|https://issues.apache.org/jira/browse/ORC-477] is merged into
branch-1.5 and orc-core v1.5.6 is released.
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: (was: HIVE-21397.patch)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
(v7.6.3
[jira] [Updated] (HIVE-15406) Consider vectorizing the new 'trunc' function
[ https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15406: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you [~abstractdog]! > Consider vectorizing the new 'trunc' function > - > > Key: HIVE-15406 > URL: https://issues.apache.org/jira/browse/HIVE-15406 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 4.0.0 >Reporter: Matt McCline >Assignee: Laszlo Bodor >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch, > HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch, > HIVE-15406.06.patch > > > Rounding function 'trunc' added by HIVE-14582. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: HIVE-21397.patch
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16255: Labels: todoc4.0 (was: ) > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch, > HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch, > HIVE-16255.06.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16255: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you [~abstractdog]! I think these new udf-s should be mentioned in the documentation; somewhere around: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-Built-inAggregateFunctions(UDAF) > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch, > HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch, > HIVE-16255.06.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219216
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:24
Start Date: 27/Mar/19 09:24
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269467626
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -2130,11 +2144,18 @@ private void create_table_core(final RawStore ms,
final Table tbl,
// If the table has column statistics, update it into the metastore.
This feature is used
// by replication to replicate table level statistics.
- if (tbl.isSetColStats()) {
-// We do not replicate statistics for a transactional table right now
and hence we do not
-// expect a transactional table to have column statistics here. So
passing null
-// validWriteIds is fine for now.
-updateTableColumnStatsInternal(tbl.getColStats(), null,
tbl.getWriteId());
+ if (colStats != null) {
+// On replica craft a valid snapshot out of the writeId in the table.
+long writeId = tbl.getWriteId();
+String validWriteIds = null;
+if (writeId > 0) {
+ ValidWriteIdList vwil =
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219216)
Time Spent: 3.5h (was: 3h 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219217
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:24
Start Date: 27/Mar/19 09:24
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269467699
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -2130,11 +2144,18 @@ private void create_table_core(final RawStore ms,
final Table tbl,
// If the table has column statistics, update it into the metastore.
This feature is used
// by replication to replicate table level statistics.
- if (tbl.isSetColStats()) {
-// We do not replicate statistics for a transactional table right now
and hence we do not
-// expect a transactional table to have column statistics here. So
passing null
-// validWriteIds is fine for now.
-updateTableColumnStatsInternal(tbl.getColStats(), null,
tbl.getWriteId());
+ if (colStats != null) {
+// On replica craft a valid snapshot out of the writeId in the table.
+long writeId = tbl.getWriteId();
+String validWriteIds = null;
+if (writeId > 0) {
+ ValidWriteIdList vwil =
+ new
ValidReaderWriteIdList(TableName.getDbTable(tbl.getDbName(),
Review comment:
Done. Please check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219217)
Time Spent: 3h 40m (was: 3.5h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Status: Patch Available (was: Open) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch, > HIVE-21516.03.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Status: Open (was: Patch Available) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch, > HIVE-21516.03.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219218&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219218
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:24
Start Date: 27/Mar/19 09:24
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269467769
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3539,10 +3573,19 @@ public boolean equals(Object obj) {
}
// Update partition column statistics if available
-for (Partition newPart : newParts) {
- if (newPart.isSetColStats()) {
-updatePartitonColStatsInternal(tbl, newPart.getColStats(), null,
newPart.getWriteId());
+int cnt = 0;
+for (ColumnStatistics partColStats: partsColStats) {
+ long writeId = partsWriteIds.get(cnt++);
+ // On replica craft a valid snapshot out of the writeId in the
partition
+ String validWriteIds = null;
+ if (writeId > 0) {
+ValidWriteIdList vwil =
Review comment:
Done. Please check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219218)
Time Spent: 3h 50m (was: 3h 40m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Attachment: HIVE-21516.03.patch > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch, > HIVE-21516.03.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15406) Consider vectorizing the new 'trunc' function
[ https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802586#comment-16802586 ] Laszlo Bodor commented on HIVE-15406: - thanks [~kgyrtkirk]! > Consider vectorizing the new 'trunc' function > - > > Key: HIVE-15406 > URL: https://issues.apache.org/jira/browse/HIVE-15406 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 4.0.0 >Reporter: Matt McCline >Assignee: Laszlo Bodor >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch, > HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch, > HIVE-15406.06.patch > > > Rounding function 'trunc' added by HIVE-14582. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802584#comment-16802584 ] Laszlo Bodor commented on HIVE-16255: - thanks [~kgyrtkirk]! > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch, > HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch, > HIVE-16255.06.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219220&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219220
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:27
Start Date: 27/Mar/19 09:27
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269468871
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2950,21 +2956,33 @@ public Partition createPartition(Table tbl,
Map partSpec) throws
int size = addPartitionDesc.getPartitionCount();
List in =
new ArrayList(size);
-AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf,
tbl, true);
long writeId;
String validWriteIdList;
-if (tableSnapshot != null && tableSnapshot.getWriteId() > 0) {
- writeId = tableSnapshot.getWriteId();
- validWriteIdList = tableSnapshot.getValidWriteIdList();
+
+// In case of replication, get the writeId from the source and use valid
write Id list
+// for replication.
+if (addPartitionDesc.getReplicationSpec() != null &&
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219220)
Time Spent: 4h (was: 3h 50m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.47.patch > Upgrade to calcite-1.19 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch, > HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch, > HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch, > HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch, > HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch, > HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch, > HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch, > HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219226&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219226
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 09:46
Start Date: 27/Mar/19 09:46
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269476213
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java
##
@@ -381,4 +382,11 @@ public void setOwnerName(String ownerName) {
throw new RuntimeException("Invalid table type : " + getDescType());
}
}
+
+ public Long getReplWriteId() {
+if (this.createTblDesc != null) {
+ return this.createTblDesc.getReplWriteId();
Review comment:
In getBaseAddPartitionDescFromPartition() where we use this function, we
don't have access to the event message. Instead we are passing the writeId
through ImportTableDesc by calling setReplWriteId(). This function just
introduces the missing getReplWriteId() method symmetric to setReplWriteId().
If we use local variable and pass it around there is a possibility that local
writeId variable can go out of sync with that in ImportTableDesc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219226)
Time Spent: 4h 10m (was: 4h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Attachment: HIVE-21231.06.patch
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Status: Open (was: Patch Available)
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Status: Patch Available (was: Open)
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802612#comment-16802612
]
Hive QA commented on HIVE-21516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963833/HIVE-21516.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16708/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16708/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16708/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963833 - PreCommit-HIVE-Build
> Fix spark downloading for q tests
> -
>
> Key: HIVE-21516
> URL: https://issues.apache.org/jira/browse/HIVE-21516
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch,
> HIVE-21516.03.patch
>
>
> Currently itests/pom.xml declares a command to generated the download script
> for spark, thus it is re-generated every time any maven command is executed
> for any sub project of itests. AS a side effect it is leaving download.sh
> files everywhere. The download.sh file is almost totally static, no need to
> recreate it every time, just requires $spark.version as a parameter.
> Also it is only working properly under linux, as it relies on the md5sum
> program which is not present in OS X. This means that if the spark tarball is
> partially downloaded on OS X, then it would never be re-downloaded. This
> should be fixed by making it work as well using md5 on OS X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[ https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21316: Attachment: HIVE-21316.05.patch > Comparision of varchar column and string literal should happen in varchar > - > > Key: HIVE-21316 > URL: https://issues.apache.org/jira/browse/HIVE-21316 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch, > HIVE-21316.03.patch, HIVE-21316.04.patch, HIVE-21316.05.patch > > > this is most probably the root cause behind HIVE-21310 as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21521) Upgrade ORC to 1.5.5
[
https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802660#comment-16802660
]
Hive QA commented on HIVE-21521:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16709/dev-support/hive-personality.sh
|
| git revision | master / a7a9f51 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16709/yetus/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16709/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade ORC to 1.5.5
>
>
> Key: HIVE-21521
> URL: https://issues.apache.org/jira/browse/HIVE-21521
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.4
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HIVE-21521.01.patch
>
>
> ORC 1.5.5 release notes:
> [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500]
> ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo
> buffer overflow([more
> details|https://issues.apache.org/jira/browse/SPARK-27107]).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219250 ] ASF GitHub Bot logged work on HIVE-21109: - Author: ASF GitHub Bot Created on: 27/Mar/19 11:11 Start Date: 27/Mar/19 11:11 Worklog Time Spent: 10m Work Description: ashutosh-bapat commented on pull request #579: HIVE-21109 : Support stats replication for ACID tables. URL: https://github.com/apache/hive/pull/579#discussion_r269508934 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableDesc.java ## @@ -118,7 +118,8 @@ List notNullConstraints; List defaultConstraints; List checkConstraints; - private ColumnStatistics colStats; + private ColumnStatistics colStats; // For the sake of replication + private long writeId = -1; // For the sake of replication Review comment: I was initially afraid that there could be other side-effects of this change. Your suggestion will bring all writeId replication through replWriteId, which is good. Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 219250) Time Spent: 4h 20m (was: 4h 10m) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21521) Upgrade ORC to 1.5.5
[
https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802677#comment-16802677
]
Hive QA commented on HIVE-21521:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963846/HIVE-21521.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16709/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16709/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16709/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963846 - PreCommit-HIVE-Build
> Upgrade ORC to 1.5.5
>
>
> Key: HIVE-21521
> URL: https://issues.apache.org/jira/browse/HIVE-21521
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.3.4
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HIVE-21521.01.patch
>
>
> ORC 1.5.5 release notes:
> [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500]
> ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo
> buffer overflow([more
> details|https://issues.apache.org/jira/browse/SPARK-27107]).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802694#comment-16802694
]
Hive QA commented on HIVE-21511:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} beeline: The patch generated 2 new + 81 unchanged - 0
fixed = 83 total (was 81) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16710/dev-support/hive-personality.sh
|
| git revision | master / a7a9f51 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16710/yetus/diff-checkstyle-beeline.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16710/yetus/patch-asflicense-problems.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16710/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219260&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219260
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 11:56
Start Date: 27/Mar/19 11:56
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269523732
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
##
@@ -2689,7 +2689,19 @@ private int alterTable(Hive db, AlterTableDesc
alterTbl) throws HiveException {
} else {
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
-boolean isTxn = alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS;
+boolean isTxn = false;
+if (alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS) {
+ // ADDPROPS is used to add repl.last.id during replication. That's
not a transactional
+ // change.
+ Map props = alterTbl.getProps();
+ if (props.size() <= 1 &&
props.get(ReplicationSpec.KEY.CURR_STATE_ID.toString()) != null) {
Review comment:
itself is wrong. I do not see any ADDPROPS usage which is updating
transactional properties. All those seem to come through AddPartition and not
alterTable for partitioned table. So, may be we can safely mark this as
non-transactional always. Does that look right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219260)
Time Spent: 4.5h (was: 4h 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219266
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 12:03
Start Date: 27/Mar/19 12:03
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269523732
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
##
@@ -2689,7 +2689,19 @@ private int alterTable(Hive db, AlterTableDesc
alterTbl) throws HiveException {
} else {
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
-boolean isTxn = alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS;
+boolean isTxn = false;
+if (alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS) {
+ // ADDPROPS is used to add repl.last.id during replication. That's
not a transactional
+ // change.
+ Map props = alterTbl.getProps();
+ if (props.size() <= 1 &&
props.get(ReplicationSpec.KEY.CURR_STATE_ID.toString()) != null) {
Review comment:
The comment
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
itself looks wrong. I do not see any ADDPROPS usage which is updating
statistics properties. All those seem to come through AddPartition and not
alterTable for partitioned table. So, may be we can safely mark this as
non-transactional always. Does that look right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219266)
Time Spent: 4h 40m (was: 4.5h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802724#comment-16802724
]
Hive QA commented on HIVE-21511:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963847/HIVE-21511.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15878 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16710/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16710/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16710/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963847 - PreCommit-HIVE-Build
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.48.patch > Upgrade to calcite-1.19 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch, > HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch, > HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch, > HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch, > HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch, > HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch, > HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch, > HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch, > HIVE-21001.48.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Status: Open (was: Patch Available)
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch,
> HIVE-21511.3.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Attachment: HIVE-21511.3.patch
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch,
> HIVE-21511.3.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-21511:
Status: Patch Available (was: Open)
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch, HIVE-21511.2.patch,
> HIVE-21511.3.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802743#comment-16802743
]
Denys Kuzmenko commented on HIVE-9995:
--
[~vgumashta] could you please review. Thank you.
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch,
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch,
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802757#comment-16802757
]
Hive QA commented on HIVE-21516:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16711/dev-support/hive-personality.sh
|
| git revision | master / a7a9f51 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16711/yetus/patch-asflicense-problems.txt
|
| modules | C: itests itests/hive-unit itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16711/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix spark downloading for q tests
> -
>
> Key: HIVE-21516
> URL: https://issues.apache.org/jira/browse/HIVE-21516
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch,
> HIVE-21516.03.patch
>
>
> Currently itests/pom.xml declares a command to generated the download script
> for spark, thus it is re-generated every time any maven command is executed
> for any sub project of itests. AS a side effect it is leaving download.sh
> files everywhere. The download.sh file is almost totally static, no need to
> recreate it every time, just requires $spark.version as a parameter.
> Also it is only working properly under linux, as it relies on the md5sum
> program which is not present in OS X. This means that if the spark tarball is
> partially downloaded on OS X, then it would never be re-downloaded. This
> should be fixed by making it work as well using md5 on OS X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[ https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21316: Attachment: HIVE-21316.06.patch > Comparision of varchar column and string literal should happen in varchar > - > > Key: HIVE-21316 > URL: https://issues.apache.org/jira/browse/HIVE-21316 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch, > HIVE-21316.03.patch, HIVE-21316.04.patch, HIVE-21316.05.patch, > HIVE-21316.06.patch > > > this is most probably the root cause behind HIVE-21310 as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21512) Upgrade jms-api to 2.0.2
[ https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21512: Attachment: HIVE-21512.01.patch > Upgrade jms-api to 2.0.2 > > > Key: HIVE-21512 > URL: https://issues.apache.org/jira/browse/HIVE-21512 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21512.01.patch, HIVE-21512.01.patch > > > I've noticed that for some time that sometimes there are issues with > javax.jms:jms:1.1 artifact - because it doesn't seem to be available from > maven central for some reason; > https://issues.sonatype.org/browse/MVNCENTRAL-4708 > Alternatively; I think we might try to just upgrade to 2.0.2 version of the > jms-api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21512) Upgrade jms-api to 2.0.2
[ https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802779#comment-16802779 ] Zoltan Haindrich commented on HIVE-21512: - [~pvary] Could you please take a look? > Upgrade jms-api to 2.0.2 > > > Key: HIVE-21512 > URL: https://issues.apache.org/jira/browse/HIVE-21512 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21512.01.patch, HIVE-21512.01.patch > > > I've noticed that for some time that sometimes there are issues with > javax.jms:jms:1.1 artifact - because it doesn't seem to be available from > maven central for some reason; > https://issues.sonatype.org/browse/MVNCENTRAL-4708 > Alternatively; I think we might try to just upgrade to 2.0.2 version of the > jms-api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Labels: pull-request-available refactor-ddl (was: pull-request-available) > Break up DDLTask - extract Database related operations > -- > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch, > HIVE-21292.15.patch, HIVE-21292.15.patch, HIVE-21292.16.patch, > HIVE-21292.17.patch > > Time Spent: 7h 50m > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21346) Refactor Hive DDL
[ https://issues.apache.org/jira/browse/HIVE-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21346: -- Labels: refactor-ddl (was: ) > Refactor Hive DDL > - > > Key: HIVE-21346 > URL: https://issues.apache.org/jira/browse/HIVE-21346 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > > Hive DDL has multiple problems: > * DDLTask is a huge class, needs to be cut to smaller pieces > * DDLSemanticAnalyzer is a huge class, needs to be cut to smaller pieces > * DDLWork has a bad structure by having a field for every possible DDL > operation > * DDLDesc subclasses are put into one package which has an endless list of > classes, should be separated by subjects > * Many DDLDesc subclasses are not immutable -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21401) Break up DDLTask - extract Table related operations
[ https://issues.apache.org/jira/browse/HIVE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21401: -- Labels: refactor-ddl (was: ) > Break up DDLTask - extract Table related operations > --- > > Key: HIVE-21401 > URL: https://issues.apache.org/jira/browse/HIVE-21401 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21401.01.patch, HIVE-21401.02.patch, > HIVE-21401.03.patch, HIVE-21401.04.patch, HIVE-21401.05.patch, > HIVE-21401.06.patch, HIVE-21401.07.patch, HIVE-21401.08.patch, > HIVE-21401.09.patch, HIVE-21401.10.patch, HIVE-21401.11.patch, > HIVE-21401.12.patch, HIVE-21401.13.patch, HIVE-21401.14.patch, > HIVE-21401.15.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #2: extract all the table related operations from the old DDLTask except > alter table, and move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21523) Break up DDLTask - extract View related operations
[ https://issues.apache.org/jira/browse/HIVE-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21523: -- Description: DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also a huge class, which has a field for each DDL operation it supports. The goal is to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: * have a separate class for each operation * have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable * make all the requests (DDLDesc subclasses) immutable * DDLTask should be agnostic to the actual operations * right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...) In the interim time when there are two DDLTask and DDLWork classes in the code base the new ones in the new package are called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes are in use. Step #3: extract all the view related operations from the old DDLTask, and move them under the new package. > Break up DDLTask - extract View related operations > -- > > Key: HIVE-21523 > URL: https://issues.apache.org/jira/browse/HIVE-21523 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #3: extract all the view related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21523) Break up DDLTask - extract View related operations
[ https://issues.apache.org/jira/browse/HIVE-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21523: - > Break up DDLTask - extract View related operations > -- > > Key: HIVE-21523 > URL: https://issues.apache.org/jira/browse/HIVE-21523 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21523) Break up DDLTask - extract View related operations
[ https://issues.apache.org/jira/browse/HIVE-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21523: -- Attachment: HIVE-21523.01.patch > Break up DDLTask - extract View related operations > -- > > Key: HIVE-21523 > URL: https://issues.apache.org/jira/browse/HIVE-21523 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21523.01.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #3: extract all the view related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21523) Break up DDLTask - extract View related operations
[ https://issues.apache.org/jira/browse/HIVE-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21523: -- Status: Patch Available (was: Open) > Break up DDLTask - extract View related operations > -- > > Key: HIVE-21523 > URL: https://issues.apache.org/jira/browse/HIVE-21523 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21523.01.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #3: extract all the view related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802818#comment-16802818
]
Hive QA commented on HIVE-21516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963851/HIVE-21516.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 806 failed/errored test(s), 15876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket4]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket5]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket6]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin6]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[empty_dir_in_table]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[external_table_with_space_in_location_path]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[gen_udf_example_add10]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_bucketed_table]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_merge]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[insert_overwrite_directory2]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[leftsemijoin_mr]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[list_bucket_dml_10]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge1]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge2]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge3]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge4]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge5]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge6]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge7]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge8]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge9]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat1]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[parallel_orderby]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[quotedid_smb]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[reduce_deduplicate]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[remote_script]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[root_dir_external_table]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority2]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCl
[jira] [Commented] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
[ https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802892#comment-16802892 ] Jesus Camacho Rodriguez commented on HIVE-21290: I had not pushed new binary files... I have pushed an addendum in a new commit. > Restore historical way of handling timestamps in Parquet while keeping the > new semantics at the same time > - > > Key: HIVE-21290 > URL: https://issues.apache.org/jira/browse/HIVE-21290 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Ivanfi >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21290.1.patch, HIVE-21290.2.patch, > HIVE-21290.2.patch, HIVE-21290.3.patch, HIVE-21290.4.patch, > HIVE-21290.4.patch, HIVE-21290.5.patch > > > This sub-task is for implementing the Parquet-specific parts of the following > plan: > h1. Problem > Historically, the semantics of the TIMESTAMP type in Hive depended on the > file format. Timestamps in Avro, Parquet and RCFiles with a binary SerDe had > _Instant_ semantics, while timestamps in ORC, textfiles and RCFiles with a > text SerDe had _LocalDateTime_ semantics. > The Hive community wanted to get rid of this inconsistency and have > _LocalDateTime_ semantics in Avro, Parquet and RCFiles with a binary SerDe as > well. *Hive 3.1 turned off normalization to UTC* to achieve this. While this > leads to the desired new semantics, it also leads to incorrect results when > new Hive versions read timestamps written by old Hive versions or when old > Hive versions or any other component not aware of this change (including > legacy Impala and Spark versions) read timestamps written by new Hive > versions. > h1. Solution > To work around this issue, Hive *should restore the practice of normalizing > to UTC* when writing timestamps to Avro, Parquet and RCFiles with a binary > SerDe. In itself, this would restore the historical _Instant_ semantics, > which is undesirable. In order to achieve the desired _LocalDateTime_ > semantics in spite of normalizing to UTC, newer Hive versions should record > the session-local local time zone in the file metadata fields serving > arbitrary key-value storage purposes. > When reading back files with this time zone metadata, newer Hive versions (or > any other new component aware of this extra metadata) can achieve > _LocalDateTime_ semantics by *converting from UTC to the saved time zone > (instead of to the local time zone)*. Legacy components that are unaware of > the new metadata can read the files without any problem and the timestamps > will show the historical Instant behaviour to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802915#comment-16802915
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963884/HIVE-21001.48.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15876 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16712/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16712/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16712/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963884 - PreCommit-HIVE-Build
> Upgrade to calcite-1.19
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch,
> HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch,
> HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch,
> HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch,
> HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch,
> HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch,
> HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch,
> HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch,
> HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch,
> HIVE-21001.48.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802923#comment-16802923
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
36s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 7 new + 303 unchanged - 45
fixed = 310 total (was 348) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
10s{color} | {color:red} root: The patch generated 7 new + 303 unchanged - 45
fixed = 310 total (was 348) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
43s{color} | {color:red} ql generated 1 new + 2256 unchanged - 0 fixed = 2257
total (was 2256) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Switch statement found in
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTBuilder.literal(RexLiteral)
where default case is missing At ASTBuilder.java:where default case is
missing At ASTBuilder.java:[lines 279-290] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16712/dev-support/hive-personality.sh
|
| git revision | master / a7a9f51 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16712/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16712/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16712/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16712/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16712/yetus/p
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802925#comment-16802925 ] Thejas M Nair commented on HIVE-21517: -- +1 cc [~vgumashta] (original developer) > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802944#comment-16802944 ] Zoltan Haindrich commented on HIVE-21001: - parquet_historical_timestamp failed because of a missing file of HIVE-21290. [~ashutoshc], [~jcamachorodriguez], [~vgarg] Could you please take a look? rb: https://reviews.apache.org/r/69918/ > Upgrade to calcite-1.19 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch, > HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch, > HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch, > HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch, > HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch, > HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch, > HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch, > HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch, > HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch, > HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch, > HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch, > HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch, > HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch, > HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch, > HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch, > HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch, > HIVE-21001.45.patch, HIVE-21001.46.patch, HIVE-21001.47.patch, > HIVE-21001.48.patch > > > XLEAR LIBRARY CACHE -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21512) Upgrade jms-api to 2.0.2
[ https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802963#comment-16802963 ] Peter Vary commented on HIVE-21512: --- +1 > Upgrade jms-api to 2.0.2 > > > Key: HIVE-21512 > URL: https://issues.apache.org/jira/browse/HIVE-21512 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21512.01.patch, HIVE-21512.01.patch > > > I've noticed that for some time that sometimes there are issues with > javax.jms:jms:1.1 artifact - because it doesn't seem to be available from > maven central for some reason; > https://issues.sonatype.org/browse/MVNCENTRAL-4708 > Alternatively; I think we might try to just upgrade to 2.0.2 version of the > jms-api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21512) Upgrade jms-api to 2.0.2
[ https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802963#comment-16802963 ] Peter Vary edited comment on HIVE-21512 at 3/27/19 3:52 PM: +1 - might want to get a green run though was (Author: pvary): +1 > Upgrade jms-api to 2.0.2 > > > Key: HIVE-21512 > URL: https://issues.apache.org/jira/browse/HIVE-21512 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21512.01.patch, HIVE-21512.01.patch > > > I've noticed that for some time that sometimes there are issues with > javax.jms:jms:1.1 artifact - because it doesn't seem to be available from > maven central for some reason; > https://issues.sonatype.org/browse/MVNCENTRAL-4708 > Alternatively; I think we might try to just upgrade to 2.0.2 version of the > jms-api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21520) Query "Submit plan" time reported is incorrect
[
https://issues.apache.org/jira/browse/HIVE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803015#comment-16803015
]
Gopal V commented on HIVE-21520:
Is it a skipped state in PerfLogger?
> Query "Submit plan" time reported is incorrect
> --
>
> Key: HIVE-21520
> URL: https://issues.apache.org/jira/browse/HIVE-21520
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Trivial
>
> Hive master branch + LLAP
> {noformat}
> Query Execution Summary
> --
> OPERATION DURATION
> --
> Compile Query 0.00s
> Prepare Plan 0.00s
> Get Query Coordinator (AM) 0.00s
> Submit Plan 1553658149.89s
> Start DAG 0.53s
> Run DAG 0.43s
> --
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803017#comment-16803017
]
Gopal V commented on HIVE-21510:
Is {{(null or dt1 is not null)}} just the same as (dt1)?
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-21510:
---
Comment: was deleted
(was: Is {{(null or dt1 is not null)}} just the same as (dt1)?)
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21525) [cosmetic] reformat code in NanoTimeUtils.java
[ https://issues.apache.org/jira/browse/HIVE-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-21525: > [cosmetic] reformat code in NanoTimeUtils.java > -- > > Key: HIVE-21525 > URL: https://issues.apache.org/jira/browse/HIVE-21525 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Fix For: 4.0.0 > > > indentation is off by 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21525) [cosmetic] reformat code in NanoTimeUtils.java
[ https://issues.apache.org/jira/browse/HIVE-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21525: - Attachment: HIVE-21525.patch Status: Patch Available (was: Open) > [cosmetic] reformat code in NanoTimeUtils.java > -- > > Key: HIVE-21525 > URL: https://issues.apache.org/jira/browse/HIVE-21525 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21525.patch > > > indentation is off by 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21525) [cosmetic] reformat code in NanoTimeUtils.java
[ https://issues.apache.org/jira/browse/HIVE-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803026#comment-16803026 ] Karen Coppage commented on HIVE-21525: -- [~jcamachorodriguez], do you care enough to commit this? :D > [cosmetic] reformat code in NanoTimeUtils.java > -- > > Key: HIVE-21525 > URL: https://issues.apache.org/jira/browse/HIVE-21525 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21525.patch > > > indentation is off by 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21525) [cosmetic] reformat code in NanoTimeUtils.java
[ https://issues.apache.org/jira/browse/HIVE-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803029#comment-16803029 ] Jesus Camacho Rodriguez commented on HIVE-21525: +1, thanks [~klcopp] > [cosmetic] reformat code in NanoTimeUtils.java > -- > > Key: HIVE-21525 > URL: https://issues.apache.org/jira/browse/HIVE-21525 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21525.patch > > > indentation is off by 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21526) JSONDropDatabaseMessage needs to have the full database object.
[ https://issues.apache.org/jira/browse/HIVE-21526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-21526: --- > JSONDropDatabaseMessage needs to have the full database object. > --- > > Key: HIVE-21526 > URL: https://issues.apache.org/jira/browse/HIVE-21526 > Project: Hive > Issue Type: Improvement >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > The metastore notification event DROP_DATABASE does not provide full-thrift > objects as of now. > We have added CREATION_TIME to databases in HIVE-21077, and metadata like > this would be useful in notification processing. One of the use-cases is > IMPALA-8338. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803053#comment-16803053
]
Hive QA commented on HIVE-21231:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
37s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 11
fixed = 16 total (was 27) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16713/dev-support/hive-personality.sh
|
| git revision | master / 368a8fe |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16713/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16713/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21521) Upgrade ORC to 1.5.5
[ https://issues.apache.org/jira/browse/HIVE-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803072#comment-16803072 ] Vaibhav Gumashta commented on HIVE-21521: - +1 [~yumwang] You might have to run precommit tests against that branch as well. > Upgrade ORC to 1.5.5 > > > Key: HIVE-21521 > URL: https://issues.apache.org/jira/browse/HIVE-21521 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.4 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > Attachments: HIVE-21521.01.patch > > > ORC 1.5.5 release notes: > [https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.5.5%22&tempMax=500] > ORC-476 Make SearchAgument kryo buffer size configurable which can avoid Kryo > buffer overflow([more > details|https://issues.apache.org/jira/browse/SPARK-27107]). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21524) Impala Engine
[
https://issues.apache.org/jira/browse/HIVE-21524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803073#comment-16803073
]
Xuefu Zhang commented on HIVE-21524:
This sounds interesting. However, I'm trying to figure out what a user gains
from this if query is simply routed to Impala coordinator. As an alternative, a
user can just connect the client (ex. beeline) to Impala coordinator directly.
> Impala Engine
> -
>
> Key: HIVE-21524
> URL: https://issues.apache.org/jira/browse/HIVE-21524
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Priority: Major
>
> Now that Impala has "dedicated coordinator" capability, it could be
> interesting to pair HiveServer2 instances with Impala dedicated coordinators
> on the same localhost. A client could request an 'impala' execution engine
> and subsequent queries would be routed to the local coordinator.
> {code:sql}
> set hive.execution.engine=impala;
> {code}
> This would allow clients seamless access to both capabilities without needing
> different connections or drivers, Hive would also be a central location for
> auditing and authorization.
> https://www.cloudera.com/documentation/enterprise/latest/topics/impala_dedicated_coordinator.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21226) Exclude read-only transactions from ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803101#comment-16803101 ] Denys Kuzmenko commented on HIVE-21226: --- [~vgumashta], [~pvary] do you know if this is still relevant? My understanding was that we are now using ValidTxnWriteIdList, so there won't be anything related to ReadOnly transactions. Do I miss something? Is there some particular scenario where we would benefit from this optimization? > Exclude read-only transactions from ValidTxnList > > > Key: HIVE-21226 > URL: https://issues.apache.org/jira/browse/HIVE-21226 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Denys Kuzmenko >Priority: Major > > Once HIVE-21114 is done, we should make sure that ValidTxnList doesn't > contain any read-only txns in the exceptions list since by definition there > is no data tagged with such txnid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21404:
Status: In Progress (was: Patch Available)
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch, HIVE-21404.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803110#comment-16803110
]
Hive QA commented on HIVE-21231:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963855/HIVE-21231.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15877 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16713/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16713/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16713/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963855 - PreCommit-HIVE-Build
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21404) MSSQL upgrade script alters the wrong column
[
https://issues.apache.org/jira/browse/HIVE-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati updated HIVE-21404:
Attachment: HIVE-21404.3.patch
Status: Patch Available (was: In Progress)
Thanks [~kgyrtkirk] for the +1, thanks [~ashutosh.bapat] for pointing out the
extra SQL Server issues.
I've submitted the final version that was able to complete the schematool
upgrade process in a similar fashion to HIVE-21462.
> MSSQL upgrade script alters the wrong column
>
>
> Key: HIVE-21404
> URL: https://issues.apache.org/jira/browse/HIVE-21404
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.2.0
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Attachments: HIVE-21404.1.patch, HIVE-21404.2.patch,
> HIVE-21404.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HIVE-20221 changes PARTITION_PARAMS, so the following command is modifying
> the wrong table:
> {{ALTER TABLE "SERDE_PARAMS" ALTER COLUMN "PARAM_VALUE" nvarchar(MAX);}}
> https://github.com/apache/hive/blob/d3b036920acde7bb04840697eb13038103b062b4/standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.1.0-to-3.2.0.mssql.sql#L21
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803116#comment-16803116
]
Miklos Gergely commented on HIVE-21231:
---
[~vgarg] we have a clean build, please take a look, and merge it if you agree.
I've slightly refactored
HiveJoinAddNotNullRule to have a cleaner structure.
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch,
> HIVE-21231.06.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803124#comment-16803124
]
Vineet Garg commented on HIVE-21230:
[~jcamachorodriguez] Can you take a look please?
https://reviews.apache.org/r/70326/
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803123#comment-16803123
]
Vihang Karajgaonkar commented on HIVE-21484:
Hi [~ngangam] Sorry for not getting back earlier. In my opinion schema version
of the db is a implementation detail which end users should not be exposed to.
So having the source code (project version) exposed it more reasonable since it
can be tied with the APIs easily in the future.
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-21518: -- Attachment: HIVE-21518.2.patch > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch, HIVE-21518.2.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803128#comment-16803128 ] Jason Dere commented on HIVE-21518: --- Accidentally added llap_notequalns.q to TestMiniLlapCliDriver rather than TestMiniLlapLocalCliDriver, fixed in patch v2. > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch, HIVE-21518.2.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803135#comment-16803135
]
Vihang Karajgaonkar commented on HIVE-21484:
The failing tests above work for me.
{noformat}
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hive.jdbc.TestActivePassiveHA
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.773 s
- in org.apache.hive.jdbc.TestActivePassiveHA
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
