[jira] [Updated] (HIVE-19727) Fix Signature matching of table aliases
[ https://issues.apache.org/jira/browse/HIVE-19727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19727: Attachment: (was: HIVE-19727.02.patch) > Fix Signature matching of table aliases > --- > > Key: HIVE-19727 > URL: https://issues.apache.org/jira/browse/HIVE-19727 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19727.01.patch, HIVE-19727.02.patch, > HIVE-19727.03.patch > > > there is a probable problem with alias matching: "t1 as a" is matched to "t2 > as a" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19727) Fix Signature matching of table aliases
[ https://issues.apache.org/jira/browse/HIVE-19727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19727: Attachment: HIVE-19727.03.patch > Fix Signature matching of table aliases > --- > > Key: HIVE-19727 > URL: https://issues.apache.org/jira/browse/HIVE-19727 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19727.01.patch, HIVE-19727.02.patch, > HIVE-19727.03.patch > > > there is a probable problem with alias matching: "t1 as a" is matched to "t2 > as a" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19727) Fix Signature matching of table aliases
[ https://issues.apache.org/jira/browse/HIVE-19727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19727: Attachment: HIVE-19727.02.patch > Fix Signature matching of table aliases > --- > > Key: HIVE-19727 > URL: https://issues.apache.org/jira/browse/HIVE-19727 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19727.01.patch, HIVE-19727.02.patch, > HIVE-19727.03.patch > > > there is a probable problem with alias matching: "t1 as a" is matched to "t2 > as a" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18079: --- Attachment: HIVE-18079.20.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.17.patch, > HIVE-18079.17.patch, HIVE-18079.18.patch, HIVE-18079.19.patch, > HIVE-18079.2.patch, HIVE-18079.20.patch, HIVE-18079.4.patch, > HIVE-18079.5.patch, HIVE-18079.6.patch, HIVE-18079.7.patch, > HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496166#comment-16496166 ] Matt McCline commented on HIVE-19529: - The triggers test failures seem unrelated and they seem like flakey tests. Rebase and resubmit same changes again as patch #6. > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.06.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19687) Export table on acid partitioned table is failing
[
https://issues.apache.org/jira/browse/HIVE-19687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496164#comment-16496164
]
Hive QA commented on HIVE-19687:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
30s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 625 unchanged - 0
fixed = 626 total (was 625) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11376/dev-support/hive-personality.sh
|
| git revision | master / cab1e60 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11376/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11376/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Export table on acid partitioned table is failing
> -
>
> Key: HIVE-19687
> URL: https://issues.apache.org/jira/browse/HIVE-19687
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19687.1.patch, HIVE-19687.2.patch,
> HIVE-19687.3.patch
>
>
> *Reproducer*
> {code:sql}
> create table exportPartitionTable(id int, name string) partitioned by(country
> string) clustered by (id) into 2 buckets stored as orc tblproperties
> ("transactional"="true");
> export table exportPartitionTable PARTITION (country='india') to
> '/tmp/exportDataStore';
> {code}
> *Error*
> {noformat}
> FAILED: SemanticException [Error 10004]: Line 1:165 Invalid table alias or
> column reference 'india': (possible column names are: id, name, country)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: (was: HIVE-19529.05.patch) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.06.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Status: In Progress (was: Patch Available) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.06.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Status: Patch Available (was: In Progress) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.06.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: HIVE-19529.06.patch > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.06.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19096) query result cache interferes with explain analyze
[
https://issues.apache.org/jira/browse/HIVE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-19096:
--
Attachment: HIVE-19096.3.patch
> query result cache interferes with explain analyze
> ---
>
> Key: HIVE-19096
> URL: https://issues.apache.org/jira/browse/HIVE-19096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-19096.1.patch, HIVE-19096.2.patch,
> HIVE-19096.3.patch
>
>
> if result cache is active; the explain analyze doesn't really return usefull
> informations; even for unseen queries the result is like this:
> {code}
> ++
> |Explain |
> ++
> | Stage-0|
> | Fetch Operator |
> | Cached Query Result:true,limit:-1 |
> ||
> ++
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster
[ https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496090#comment-16496090 ] Sankar Hariappan edited comment on HIVE-19340 at 5/31/18 6:04 AM: -- [~ekoifman], [~sershe], Currently, user/admin have the privilege to abort the replicated txn if they need to kill it explicitly. Allowing timeout on replicated txn is not ideal as several factors such as network delay, source cluster down, replication slow down etc can cause timeout and this is indeterministic value. If the source melts down with a txn which later comes alive and committed but it is aborted at target by admin/timeout. If this committed txn is propagated to target, it will be ignored causing loss of data. In fact this is a use-case for fail-over where the source shouldn't propagate any operations after shutdown and so replication need not handle it. Also, any manual changes by admin at target shall affect replication which we don't handle. So, we expect admin to abort replicated txn only if they knew source is no longer available and fail-over is triggered. cc [~maheshk114], [~thejas], [~anishek] was (Author: sankarh): [~ekoifman], [~sershe], Currently, user/admin have the privilege to abort the replicated txn if they need to kill it explicitly. Allowing timeout on replicated txn is not ideal as several factors such as network delay, source cluster down, Beacon slow down etc can cause timeout and this is indeterministic value. Also, if the txn comes alive and committed at source but it is aborted in target, and if committed txn is propagated to target, it will be ignored at target causing loss of data. I think, this use-case should be handled by replication. cc [~maheshk114], [~thejas], [~anishek] > Disable timeout of transactions opened by replication task at target cluster > > > Key: HIVE-19340 > URL: https://issues.apache.org/jira/browse/HIVE-19340 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-19340.01.patch, HIVE-19340.02.patch, > HIVE-19340.03-branch-3.patch, HIVE-19340.03.patch, > HIVE-19340.04-branch-3.patch > > > The transactions opened by applying EVENT_OPEN_TXN should never be aborted > automatically due to time-out. Aborting of transaction started by replication > task may leads to inconsistent state at target which needs additional > overhead to clean-up. So, it is proposed to mark the transactions opened by > replication task as special ones and shouldn't be aborted if heart beat is > lost. This helps to ensure all ABORT and COMMIT events will always find the > corresponding txn at target to operate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: HIVE-19529.05.patch > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.05.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[
https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496142#comment-16496142
]
Hive QA commented on HIVE-19529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925814/HIVE-19529.05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14426 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesWrite
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11375/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11375/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11375/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925814 - PreCommit-HIVE-Build
> Vectorization: Date/Timestamp NULL issues
> -
>
> Key: HIVE-19529
> URL: https://issues.apache.org/jira/browse/HIVE-19529
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Blocker
>
> Wrong results found for:
> date_add/date_sub
> UT areas:
> date_add/date_sub
> datediff
> to_date
> interval_year_month + interval_year_month
> interval_day_time + interval_day_time
> interval_day_time + timestamp
> timestamp + interval_day_time
> date + interval_day_time
> interval_day_time + date
> interval_year_month + date
> date + interval_year_month
> interval_year_month + interval_year_month
> timestamp + interval_year_month
> date - date
> interval_year_month - interval_year_month
> interval_day_time - interval_day_time
> timestamp - interval_day_time
> timestamp - timestamp
> date - timestamp
> timestamp - date
> date - interval_day_time
> date - interval_year_month
> timestamp - interval_year_month
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19334) Use actual file size rather than stats for fetch task optimization with external tables
[ https://issues.apache.org/jira/browse/HIVE-19334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19334: -- Attachment: HIVE-19334.4.patch > Use actual file size rather than stats for fetch task optimization with > external tables > --- > > Key: HIVE-19334 > URL: https://issues.apache.org/jira/browse/HIVE-19334 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19334.1.patch, HIVE-19334.2.patch, > HIVE-19334.3.patch, HIVE-19334.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: (was: HIVE-19529.04.patch) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: (was: HIVE-19529.05.patch) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19332) Disable compute.query.using.stats for external table
[ https://issues.apache.org/jira/browse/HIVE-19332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19332: -- Attachment: HIVE-19332.7.patch > Disable compute.query.using.stats for external table > > > Key: HIVE-19332 > URL: https://issues.apache.org/jira/browse/HIVE-19332 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19332.1.patch, HIVE-19332.2.patch, > HIVE-19332.3.patch, HIVE-19332.4.patch, HIVE-19332.5.patch, > HIVE-19332.6.patch, HIVE-19332.7.patch > > > Hive can use statistics to answer queries like count(*). This can be > problematic on external tables where another tool might add files that Hive > doesn’t know about. In that case Hive will return incorrect results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[
https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496115#comment-16496115
]
Hive QA commented on HIVE-19529:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
48s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} ql: The patch generated 224 new + 3292 unchanged - 5
fixed = 3516 total (was 3297) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
49s{color} | {color:red} ql generated 8 new + 2270 unchanged - 63 fixed = 2278
total (was 2333) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnBetween.left At
FilterCharColumnBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnBetween.left At
FilterCharColumnBetween.java:[line 44] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnBetween.right At
FilterCharColumnBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnBetween.right At
FilterCharColumnBetween.java:[line 45] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnNotBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnNotBetween.left At
FilterCharColumnNotBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnNotBetween.left At
FilterCharColumnNotBetween.java:[line 44] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnNotBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnNotBetween.right At
FilterCharColumnNotBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnNotBetween.right At
FilterCharColumnNotBetween.java:[line 45] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterVarCharColumnBetween(int,
byte[], byt
[jira] [Commented] (HIVE-19493) VectorUDFDateDiffColCol copySelected does not handle nulls correctly
[
https://issues.apache.org/jira/browse/HIVE-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496096#comment-16496096
]
Matt McCline commented on HIVE-19493:
-
[~vihangk1] I'm hoping HIVE-19529 will go in the next few days after it gets
through the long Hive QA queue for master and branch-3. Once it does I will
update and submit your patch with the @Ignore annotations removed.
> VectorUDFDateDiffColCol copySelected does not handle nulls correctly
>
>
> Key: HIVE-19493
> URL: https://issues.apache.org/jira/browse/HIVE-19493
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19493.01.patch, HIVE-19493.02.patch
>
>
> The {{copySelected}} method in {{VectorUDFDateDiffColCol}} class was missed
> during HIVE-18622
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19340) Disable timeout of transactions opened by replication task at target cluster
[ https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496090#comment-16496090 ] Sankar Hariappan commented on HIVE-19340: - [~ekoifman], [~sershe], Currently, user/admin have the privilege to abort the replicated txn if they need to kill it explicitly. Allowing timeout on replicated txn is not ideal as several factors such as network delay, source cluster down, Beacon slow down etc can cause timeout and this is indeterministic value. Also, if the txn comes alive and committed at source but it is aborted in target, and if committed txn is propagated to target, it will be ignored at target causing loss of data. I think, this use-case should be handled by replication. cc [~maheshk114], [~thejas], [~anishek] > Disable timeout of transactions opened by replication task at target cluster > > > Key: HIVE-19340 > URL: https://issues.apache.org/jira/browse/HIVE-19340 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-19340.01.patch, HIVE-19340.02.patch, > HIVE-19340.03-branch-3.patch, HIVE-19340.03.patch, > HIVE-19340.04-branch-3.patch > > > The transactions opened by applying EVENT_OPEN_TXN should never be aborted > automatically due to time-out. Aborting of transaction started by replication > task may leads to inconsistent state at target which needs additional > overhead to clean-up. So, it is proposed to mark the transactions opened by > replication task as special ones and shouldn't be aborted if heart beat is > lost. This helps to ensure all ABORT and COMMIT events will always find the > corresponding txn at target to operate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19649) Clean up inputs in JDBC PreparedStatement. Add unit tests.
[
https://issues.apache.org/jira/browse/HIVE-19649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496083#comment-16496083
]
Hive QA commented on HIVE-19649:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925543/HIVE-19649.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14421 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11374/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11374/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11374/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925543 - PreCommit-HIVE-Build
> Clean up inputs in JDBC PreparedStatement. Add unit tests.
> --
>
> Key: HIVE-19649
> URL: https://issues.apache.org/jira/browse/HIVE-19649
> Project: Hive
> Issue Type: Test
>Reporter: Mykhailo Kysliuk
>Assignee: Mykhailo Kysliuk
>Priority: Minor
> Attachments: HIVE-19649.01.patch, HIVE-19649.02.patch
>
>
> Add unit tests for feature that was implemented in
> [HIVE-18788|https://issues.apache.org/jira/browse/HIVE-18788].
> The integration tests are present, but it will be useful to catch errors
> during module build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.
[ https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496076#comment-16496076 ] Sankar Hariappan commented on HIVE-19499: - Test failures are irrelevant to the patch and is seen failing for previous ptest builds in branch-3. Patch 02-branch-3.patch is committed to branch-3. > Bootstrap REPL LOAD shall add tasks to create checkpoints for > db/tables/partitions. > --- > > Key: HIVE-19499 > URL: https://issues.apache.org/jira/browse/HIVE-19499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19499.01-branch-3.patch, HIVE-19499.01.patch, > HIVE-19499.02-branch-3.patch, HIVE-19499.02.patch > > > Currently. bootstrap REPL LOAD expect the target database to be empty or not > exist to start bootstrap load. > But, this adds overhead when there is a failure in between bootstrap load and > there is no way to resume it from where it fails. So, it is needed to create > checkpoints in table/partitions to skip the completely loaded objects. > Use the fully qualified path of the dump directory as a checkpoint > identifier. This should be added to the table / partition properties in hive > via a task, as the last task in the DAG for table / partition creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.
[ https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19499: Fix Version/s: 3.1.0 > Bootstrap REPL LOAD shall add tasks to create checkpoints for > db/tables/partitions. > --- > > Key: HIVE-19499 > URL: https://issues.apache.org/jira/browse/HIVE-19499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19499.01-branch-3.patch, HIVE-19499.01.patch, > HIVE-19499.02-branch-3.patch, HIVE-19499.02.patch > > > Currently. bootstrap REPL LOAD expect the target database to be empty or not > exist to start bootstrap load. > But, this adds overhead when there is a failure in between bootstrap load and > there is no way to resume it from where it fails. So, it is needed to create > checkpoints in table/partitions to skip the completely loaded objects. > Use the fully qualified path of the dump directory as a checkpoint > identifier. This should be added to the table / partition properties in hive > via a task, as the last task in the DAG for table / partition creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.
[ https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19499: Resolution: Fixed Target Version/s: 3.1.0, 4.0.0 (was: 3.0.0, 3.1.0, 4.0.0) Status: Resolved (was: Patch Available) > Bootstrap REPL LOAD shall add tasks to create checkpoints for > db/tables/partitions. > --- > > Key: HIVE-19499 > URL: https://issues.apache.org/jira/browse/HIVE-19499 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19499.01-branch-3.patch, HIVE-19499.01.patch, > HIVE-19499.02-branch-3.patch, HIVE-19499.02.patch > > > Currently. bootstrap REPL LOAD expect the target database to be empty or not > exist to start bootstrap load. > But, this adds overhead when there is a failure in between bootstrap load and > there is no way to resume it from where it fails. So, it is needed to create > checkpoints in table/partitions to skip the completely loaded objects. > Use the fully qualified path of the dump directory as a checkpoint > identifier. This should be added to the table / partition properties in hive > via a task, as the last task in the DAG for table / partition creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19649) Clean up inputs in JDBC PreparedStatement. Add unit tests.
[
https://issues.apache.org/jira/browse/HIVE-19649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496050#comment-16496050
]
Hive QA commented on HIVE-19649:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} jdbc: The patch generated 0 new + 1 unchanged - 94
fixed = 1 total (was 95) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11374/dev-support/hive-personality.sh
|
| git revision | master / cab1e60 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11374/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up inputs in JDBC PreparedStatement. Add unit tests.
> --
>
> Key: HIVE-19649
> URL: https://issues.apache.org/jira/browse/HIVE-19649
> Project: Hive
> Issue Type: Test
>Reporter: Mykhailo Kysliuk
>Assignee: Mykhailo Kysliuk
>Priority: Minor
> Attachments: HIVE-19649.01.patch, HIVE-19649.02.patch
>
>
> Add unit tests for feature that was implemented in
> [HIVE-18788|https://issues.apache.org/jira/browse/HIVE-18788].
> The integration tests are present, but it will be useful to catch errors
> during module build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19605) TAB_COL_STATS table has no index on db/table name
[ https://issues.apache.org/jira/browse/HIVE-19605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496046#comment-16496046 ] Vihang Karajgaonkar commented on HIVE-19605: Hi [~ngangam] Can you please take a look? > TAB_COL_STATS table has no index on db/table name > - > > Key: HIVE-19605 > URL: https://issues.apache.org/jira/browse/HIVE-19605 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Todd Lipcon >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-19605.01.patch > > > The TAB_COL_STATS table is missing an index on (CAT_NAME, DB_NAME, > TABLE_NAME). The getTableColumnStatistics call queries based on this tuple. > This makes those queries take a significant amount of time in large > metastores since they do a full table scan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19727) Fix Signature matching of table aliases
[
https://issues.apache.org/jira/browse/HIVE-19727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496042#comment-16496042
]
Hive QA commented on HIVE-19727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925681/HIVE-19727.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 14419 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=184)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join22]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union10]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union11]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union15]
(batchId=149)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union20]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union25]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union2]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union4]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union5]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union9]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_ppr]
(batchId=116)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11373/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11373/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11373/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925681 - PreCommit-HIVE-Build
> Fix Signature matching of table aliases
> ---
>
> Key: HIVE-19727
> URL: https://issues.apache.org/jira/browse/HIVE-19727
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19727.01.patch, HIVE-19727.02.patch
>
>
> there is a probable problem with alias matching: "t1 as a" is matched to "t2
> as a"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19708) Repl copy retrying with cm path even if the failure is due to network issue
[ https://issues.apache.org/jira/browse/HIVE-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19708: --- Attachment: HIVE-19708.04.patch > Repl copy retrying with cm path even if the failure is due to network issue > --- > > Key: HIVE-19708 > URL: https://issues.apache.org/jira/browse/HIVE-19708 > Project: Hive > Issue Type: Task > Components: Hive, HiveServer2, repl >Affects Versions: 3.1.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-19708.01.patch, HIVE-19708.02.patch, > HIVE-19708.04.patch > > > * During repl load > ** for filesystem based copying of file if the copy fails due to a > connection error to source Name Node, we should recreate the filesystem > object. > ** the retry logic for local file copy should be triggered using the > original source file path ( and not the CM root path ) since failure can be > due to network issues between DFSClient and NN. > * When listing files in tables / partition to include them in _files, we > should add retry logic when failure occurs. FileSystem object here also > should be recreated since the existing one might be in inconsistent state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496033#comment-16496033
]
Gopal V commented on HIVE-17896:
[~teddy.choi]: the pushdown is too aggressive at the moment.
The operator impl looks good, so it is the optimizer cases which I'm concerned
about and would like you to split the multi-op push-down into a different patch.
The best way to get the operator level changes committed right now would be to
restrict the push-down to a simple case & let it bake-through for a few test
cycles (specifically, the pushdown through GBY case in the JIRA: GBY->RS(TopN)
=> TNK->GBY->RS).
For example.
{code}
tez/topnkey.q.out
SELECT src1.key, src2.value FROM src src1 JOIN src src2 ON (src1.key =
src2.key) ORDER BY src1.key LIMIT 5
{code}
produces
{code}
+ <-Reducer 2 [SIMPLE_EDGE]
+SHUFFLE [RS_10]
+ Select Operator [SEL_9] (rows=809 width=178)
+Output:["_col0","_col1"]
+Top N Key Operator [TNK_18] (rows=809 width=178)
+ keys:_col0,sort order:+,top n:5
+ Merge Join Operator [MERGEJOIN_21] (rows=809 width=178)
+Conds:RS_6._col0=RS_7._col0(Inner),Output:["_col0","_col2"]
+ <-Map 1 [SIMPLE_EDGE]
+SHUFFLE [RS_6]
+ PartitionCols:_col0
+ Select Operator [SEL_2] (rows=500 width=87)
...
+Filter Operator [FIL_16] (rows=500 width=87)
+ predicate:key is not null
+ Top N Key Operator [TNK_19] (rows=500 width=87)
+keys:key,sort order:+,top n:5
+TableScan [TS_0] (rows=500 width=87)
+
default@src,src1,Tbl:COMPLETE,Col:COMPLETE,Output:["key"]
{code}
The TNK_18 is good, but the TNK_19 is likely to give incorrect results due to
the nature of the join filter case.
I'm happy to review these as two changes (i.e new operator & aggressive
push-down) and get them in one by one.
> TopNKey: Create a standalone vectorizable TopNKey operator
> --
>
> Key: HIVE-17896
> URL: https://issues.apache.org/jira/browse/HIVE-17896
> Project: Hive
> Issue Type: New Feature
> Components: Operators
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch,
> HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch,
> HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch
>
>
> For TPC-DS Query27, the TopN operation is delayed by the group-by - the
> group-by operator buffers up all the rows before discarding the 99% of the
> rows in the TopN Hash within the ReduceSink Operator.
> The RS TopN operator is very restrictive as it only supports doing the
> filtering on the shuffle keys, but it is better to do this before breaking
> the vectors into rows and losing the isRepeating properties.
> Adding a TopN Key operator in the physical operator tree allows the following
> to happen.
> GBY->RS(Top=1)
> can become
> TNK(1)->GBY->RS(Top=1)
> So that, the TopNKey can remove rows before they are buffered into the GBY
> and consume memory.
> Here's the equivalent implementation in Presto
> https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35
> Adding this as a sub-feature of GroupBy prevents further optimizations if the
> GBY is on keys "a,b,c" and the TopNKey is on just "a".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496032#comment-16496032
]
Ganesha Shreedhara commented on HIVE-14557:
---
[~nemon] [~lirui] Looks like the test result of
runtime_skewjoin_mapjoin_spark.q requires correction as per the change.
Handling skew join is not required if map join optimization can be done.
> Nullpointer When both SkewJoin and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Priority: Major
> Attachments: HIVE-14557.patch
>
>
> The following sql failed with return code 2 on mr.
> {noformat}
> create table a(id int,id1 int);
> create table b(id int,id1 int);
> create table c(id int,id1 int);
> set hive.optimize.skewjoin=true;
> select a.id,b.id,c.id1 from a,b,c where a.id=b.id and a.id1=c.id1;
> {noformat}
> Error log as follows:
> {noformat}
> 2016-08-17 21:13:42,081 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
> Id =0
>
> Id =21
>
> Id =28
>
> Id =16
>
> <\Children>
> Id = 28 null<\Parent>
> <\FS>
> <\Children>
> Id = 21 nullId = 33
> Id =33
> null
> <\Children>
> <\Parent>
> <\HASHTABLEDUMMY><\Parent>
> <\MAPJOIN>
> <\Children>
> Id = 0 null<\Parent>
> <\TS>
> <\Children>
> <\Parent>
> <\MAP>
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[21]
> 2016-08-17 21:13:42,084 INFO [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Initializing dummy operator
> 2016-08-17 21:13:42,086 INFO [main]
> org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0,
> RECORDS_IN:0,
> 2016-08-17 21:13:42,087 ERROR [main]
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: Hit error while closing
> operators - failing tree
> 2016-08-17 21:13:42,088 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:682)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:696)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19675) Cast to timestamps on Druid time column leads to an exception
[
https://issues.apache.org/jira/browse/HIVE-19675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19675:
---
Fix Version/s: 3.1.0
> Cast to timestamps on Druid time column leads to an exception
> -
>
> Key: HIVE-19675
> URL: https://issues.apache.org/jira/browse/HIVE-19675
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19675.2.patch, HIVE-19675.3.patch,
> HIVE-19675.4.patch, HIVE-19675.5.patch, HIVE-19675.patch
>
>
> The following query fail due to a formatting issue.
> {code}
> SELECT CAST(`ssb_druid_100`.`__time` AS TIMESTAMP) AS `x_time`,
> . . . . . . . . . . . . . . . .> SUM(`ssb_druid_100`.`lo_revenue`) AS
> `sum_lo_revenue_ok`
> . . . . . . . . . . . . . . . .> FROM `druid_ssb`.`ssb_druid_100`
> `ssb_druid_100`
> . . . . . . . . . . . . . . . .> GROUP BY CAST(`ssb_druid_100`.`__time` AS
> TIMESTAMP);
> {code}
> Exception
> {code}
> Error: java.io.IOException: java.lang.NumberFormatException: For input
> string: "1991-12-31 19:00:00" (state=,code=0)
> {code}
> [~jcamachorodriguez] maybe this is fixed by your upcoming patches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19675) Cast to timestamps on Druid time column leads to an exception
[
https://issues.apache.org/jira/browse/HIVE-19675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496018#comment-16496018
]
Vineet Garg commented on HIVE-19675:
Pushed to branch-3
> Cast to timestamps on Druid time column leads to an exception
> -
>
> Key: HIVE-19675
> URL: https://issues.apache.org/jira/browse/HIVE-19675
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19675.2.patch, HIVE-19675.3.patch,
> HIVE-19675.4.patch, HIVE-19675.5.patch, HIVE-19675.patch
>
>
> The following query fail due to a formatting issue.
> {code}
> SELECT CAST(`ssb_druid_100`.`__time` AS TIMESTAMP) AS `x_time`,
> . . . . . . . . . . . . . . . .> SUM(`ssb_druid_100`.`lo_revenue`) AS
> `sum_lo_revenue_ok`
> . . . . . . . . . . . . . . . .> FROM `druid_ssb`.`ssb_druid_100`
> `ssb_druid_100`
> . . . . . . . . . . . . . . . .> GROUP BY CAST(`ssb_druid_100`.`__time` AS
> TIMESTAMP);
> {code}
> Exception
> {code}
> Error: java.io.IOException: java.lang.NumberFormatException: For input
> string: "1991-12-31 19:00:00" (state=,code=0)
> {code}
> [~jcamachorodriguez] maybe this is fixed by your upcoming patches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19727) Fix Signature matching of table aliases
[
https://issues.apache.org/jira/browse/HIVE-19727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496008#comment-16496008
]
Hive QA commented on HIVE-19727:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} ql: The patch generated 0 new + 10 unchanged - 7
fixed = 10 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11373/dev-support/hive-personality.sh
|
| git revision | master / cab1e60 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11373/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix Signature matching of table aliases
> ---
>
> Key: HIVE-19727
> URL: https://issues.apache.org/jira/browse/HIVE-19727
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19727.01.patch, HIVE-19727.02.patch
>
>
> there is a probable problem with alias matching: "t1 as a" is matched to "t2
> as a"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496005#comment-16496005 ] Teddy Choi commented on HIVE-17896: --- [~gopalv], [~mmccline] could you review this? I rebased it on the latest master branch. > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch, > HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17896: -- Attachment: HIVE-17896.9.patch > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch, > HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496000#comment-16496000 ] Sergey Shelukhin commented on HIVE-19418: - Removed string concatenation > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.04.patch, > HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19418: Attachment: HIVE-19418.04.patch > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.04.patch, > HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19728) beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set hive.aux.jars.path
[
https://issues.apache.org/jira/browse/HIVE-19728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495991#comment-16495991
]
Hive QA commented on HIVE-19728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925535/HIVE-19728.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14419 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11372/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11372/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11372/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925535 - PreCommit-HIVE-Build
> beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set
> hive.aux.jars.path
> -
>
> Key: HIVE-19728
> URL: https://issues.apache.org/jira/browse/HIVE-19728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-19728.1.patch
>
>
> Since HIVE-19385 it's possible to redirect bin/hive to beeline. This is not
> working as expected though, because in {{bin/hive}} we're setting
> {{hive.aux.jars.path}}. This leads to the following error:
> {code}
> $ USE_BEELINE_FOR_HIVE_CLI=true hive
> ...
> Error: Could not open client transport for any of the Server URI's in
> ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException:
> Cannot modify hive.aux.jars.path at runtime. It is not in list of params that
> are allowed to be modified at runtime (state=08S01,code=0)
> Beeline version 3.0.0 by Apache Hive
> beeline>
> {code}
> We already avoid setting {{hive.aux.jars.path}} when running {{beeline}}
> service but the USE_BEELINE_FOR_HIVE_CLI override happens after that.
> I'd suggest checking the value of USE_BEELINE_FOR_HIVE_CLI right after we've
> selected the service to run (cli/beeline/...) and override cli->beeline there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19728) beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set hive.aux.jars.path
[
https://issues.apache.org/jira/browse/HIVE-19728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495956#comment-16495956
]
Hive QA commented on HIVE-19728:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11372/dev-support/hive-personality.sh
|
| git revision | master / cab1e60 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11372/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set
> hive.aux.jars.path
> -
>
> Key: HIVE-19728
> URL: https://issues.apache.org/jira/browse/HIVE-19728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-19728.1.patch
>
>
> Since HIVE-19385 it's possible to redirect bin/hive to beeline. This is not
> working as expected though, because in {{bin/hive}} we're setting
> {{hive.aux.jars.path}}. This leads to the following error:
> {code}
> $ USE_BEELINE_FOR_HIVE_CLI=true hive
> ...
> Error: Could not open client transport for any of the Server URI's in
> ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException:
> Cannot modify hive.aux.jars.path at runtime. It is not in list of params that
> are allowed to be modified at runtime (state=08S01,code=0)
> Beeline version 3.0.0 by Apache Hive
> beeline>
> {code}
> We already avoid setting {{hive.aux.jars.path}} when running {{beeline}}
> service but the USE_BEELINE_FOR_HIVE_CLI override happens after that.
> I'd suggest checking the value of USE_BEELINE_FOR_HIVE_CLI right after we've
> selected the service to run (cli/beeline/...) and override cli->beeline there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495953#comment-16495953
]
Hive QA commented on HIVE-19326:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925529/HIVE-19326.06wip05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11371/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11371/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12925529/HIVE-19326.06wip05.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925529 - PreCommit-HIVE-Build
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch, HIVE-19326.04.patch, HIVE-19326.05.patch,
> HIVE-19326.06wip01.patch, HIVE-19326.06wip02.patch, HIVE-19326.06wip03.patch,
> HIVE-19326.06wip04.patch, HIVE-19326.06wip05.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19718) Adding partitions in bulk also fetches table for each partition
[
https://issues.apache.org/jira/browse/HIVE-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495952#comment-16495952
]
Hive QA commented on HIVE-19718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925527/HIVE-19718.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 529 failed/errored test(s), 9521 tests
executed
*Failed tests:*
{noformat}
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestAvroHCatLoader - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestE2EScenarios - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestHCatLoaderEncryption - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestHCatStorerMulti - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestHCatStorerWrapper - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=213)
TestMetaStoreConnectionUrlHook - did not produce a TEST-*.xml file (likely
timed out) (batchId=213)
TestMetaStoreEndFunctionListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=213)
TestMetastoreConf - did not produce a TEST-*.xml file (likely timed out)
(batchId=213)
TestMetrics - did not produce a TEST-*.xml file (likely timed out) (batchId=213)
TestOrcHCatLoader - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestParquetHCatStorer - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
(likely timed out) (batchId=213)
TestPigHCatUtil - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestRCFileHCatLoader - did not produce a TEST-*.xml file (likely timed out)
(batchId=198)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=233)
TestSchemaToolForMetastore - did not produce a TEST-*.xml file (likely timed
out) (batchId=213)
TestStats - did not produce a TEST-*.xml file (likely timed out) (batchId=213)
TestUtil - did not produce a TEST-*.xml file (likely timed out) (batchId=198)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=255)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_database]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=258)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
(batchId=258)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.ha
[jira] [Commented] (HIVE-19718) Adding partitions in bulk also fetches table for each partition
[
https://issues.apache.org/jira/browse/HIVE-19718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495939#comment-16495939
]
Hive QA commented on HIVE-19718:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
46s{color} | {color:blue} standalone-metastore in master has 215 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore: The patch generated 4 new + 962
unchanged - 3 fixed = 966 total (was 965) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} standalone-metastore generated 0 new + 214 unchanged
- 1 fixed = 214 total (was 215) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11370/dev-support/hive-personality.sh
|
| git revision | master / cab1e60 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11370/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11370/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: itests/hcatalog-unit standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11370/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adding partitions in bulk also fetches table for each partition
> ---
>
> Key: HIVE-19718
> URL: https://issues.apache.org/jira/browse/HIVE-19718
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
> Environment: Looking at {{convertToMPart}}:
> {code:Java}
> private MPartition convertToMPart(Partition part, boolean useTableCD)
> throws InvalidObjectException, MetaException {
> MTable mt = getMTable(part.getCatName(), part.getDbName(),
> part.getTableName());
> ...
> {code}
> So what we have as a result is that we fetch table for every partition where
> it should be done just once.
>Reporter: Alexander Kolbasov
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19718.patch
>
>
> The ObjectStore.addPartitions() method does
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495920#comment-16495920
]
Hive QA commented on HIVE-19326:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925529/HIVE-19326.06wip05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 14414 tests
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=247)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=247)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_single_sourced_multi_insert]
(batchId=104)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=168)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925529 - PreCommit-HIVE-Build
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch, HIVE-19326.04.patch, HIVE-19326.05.patch,
> HIVE-19326.06wip01.patch, HIVE-19326.06wip02.patch, HIVE-19326.06wip03.patch,
> HIVE-19326.06wip04.patch, HIVE-19326.06wip05.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495903#comment-16495903
]
Hive QA commented on HIVE-19326:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/util in master has 55 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
44s{color} | {color:red} root: The patch generated 1 new + 355 unchanged - 18
fixed = 356 total (was 373) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 327 unchanged - 18
fixed = 328 total (was 345) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11368/dev-support/hive-personality.sh
|
| git revision | master / 53fda73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11368/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11368/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11368/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11368/yetus/whitespace-tabs.txt
|
| modules | C: . itests itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11368/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
[jira] [Commented] (HIVE-19728) beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set hive.aux.jars.path
[
https://issues.apache.org/jira/browse/HIVE-19728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495867#comment-16495867
]
Vaibhav Gumashta commented on HIVE-19728:
-
+1
> beeline with USE_BEELINE_FOR_HIVE_CLI fails when trying to set
> hive.aux.jars.path
> -
>
> Key: HIVE-19728
> URL: https://issues.apache.org/jira/browse/HIVE-19728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-19728.1.patch
>
>
> Since HIVE-19385 it's possible to redirect bin/hive to beeline. This is not
> working as expected though, because in {{bin/hive}} we're setting
> {{hive.aux.jars.path}}. This leads to the following error:
> {code}
> $ USE_BEELINE_FOR_HIVE_CLI=true hive
> ...
> Error: Could not open client transport for any of the Server URI's in
> ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException:
> Cannot modify hive.aux.jars.path at runtime. It is not in list of params that
> are allowed to be modified at runtime (state=08S01,code=0)
> Beeline version 3.0.0 by Apache Hive
> beeline>
> {code}
> We already avoid setting {{hive.aux.jars.path}} when running {{beeline}}
> service but the USE_BEELINE_FOR_HIVE_CLI override happens after that.
> I'd suggest checking the value of USE_BEELINE_FOR_HIVE_CLI right after we've
> selected the service to run (cli/beeline/...) and override cli->beeline there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19682:
-
Resolution: Fixed
Fix Version/s: 4.0.0
3.1.0
Status: Resolved (was: Patch Available)
Committed to master and branch-3. Thanks [~ewohlstadter] for the patch!
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch,
> HIVE-19682.3.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-17852: Attachment: HIVE-17852.05.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-18862: Fix Version/s: (was: 3.1.0) 4.0.0 > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch, > HIVE-18862.10-branch-3.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495840#comment-16495840 ] Laszlo Bodor commented on HIVE-18862: - In the meantime branch-3 went quite far from master, I'm not sure if porting to branch-3 is worth the effort (>100 conflicts). Closing this ticket. > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch, > HIVE-18862.10-branch-3.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-18862: Resolution: Fixed Status: Resolved (was: Patch Available) > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch, > HIVE-18862.10-branch-3.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan
[
https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495836#comment-16495836
]
Hive QA commented on HIVE-19237:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925804/HIVE-19237.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14419 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=172)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925804 - PreCommit-HIVE-Build
> Only use an operatorId once in a plan
> -
>
> Key: HIVE-19237
> URL: https://issues.apache.org/jira/browse/HIVE-19237
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19237.01.patch, HIVE-19237.02.patch,
> HIVE-19237.03.patch, HIVE-19237.04.patch, HIVE-19237.05.patch,
> HIVE-19237.05.patch, HIVE-19237.06.patch, HIVE-19237.07.patch,
> HIVE-19237.08.patch, HIVE-19237.08.patch
>
>
> Column stats autogather plan part is added from a plan compiled by the driver
> itself; however that driver starts to use operatorIds from 1 ; so it's
> possible that 2 SEL_1 operators end up in the same plan...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan
[
https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495825#comment-16495825
]
Hive QA commented on HIVE-19237:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
43s{color} | {color:red} root: The patch generated 1 new + 189 unchanged - 2
fixed = 190 total (was 191) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 189 unchanged - 2
fixed = 190 total (was 191) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11367/dev-support/hive-personality.sh
|
| git revision | master / 53fda73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11367/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11367/yetus/diff-checkstyle-ql.txt
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11367/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Only use an operatorId once in a plan
> -
>
> Key: HIVE-19237
> URL: https://issues.apache.org/jira/browse/HIVE-19237
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19237.01.patch, HIVE-19237.02.patch,
> HIVE-19237.03.patch, HIVE-19237.04.patch, HIVE-19237.05.patch,
> HIVE-19237.05.patch, HIVE-19237.06.patch, HIVE-19237.07.patch,
> HIVE-19237.08.patch, HIVE-19237.08.patch
>
>
> Column stats autogather plan part is added from a plan compiled by the driver
> itself; however that driver starts to use operatorIds from 1 ; so it's
> possible that 2 SEL_1 operators end up in the same pla
[jira] [Updated] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18792:
---
Status: Open (was: Patch Available)
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-18792.1.patch, HIVE-18792.2.patch,
> HIVE-18792.3.patch
>
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18792:
---
Status: Patch Available (was: Open)
Patch(3) fixes test failure.
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-18792.1.patch, HIVE-18792.2.patch,
> HIVE-18792.3.patch
>
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18792:
---
Attachment: HIVE-18792.3.patch
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-18792.1.patch, HIVE-18792.2.patch,
> HIVE-18792.3.patch
>
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18973) Make transaction system work with catalogs
[ https://issues.apache.org/jira/browse/HIVE-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-18973: -- Status: Patch Available (was: Open) > Make transaction system work with catalogs > -- > > Key: HIVE-18973 > URL: https://issues.apache.org/jira/browse/HIVE-18973 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18973.patch > > > The transaction tables need to understand catalogs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18973) Make transaction system work with catalogs
[ https://issues.apache.org/jira/browse/HIVE-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-18973: -- Attachment: HIVE-18973.patch > Make transaction system work with catalogs > -- > > Key: HIVE-18973 > URL: https://issues.apache.org/jira/browse/HIVE-18973 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18973.patch > > > The transaction tables need to understand catalogs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: HIVE-16295.6.patch
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch,
> HIVE-16295.3.WIP.patch, HIVE-16295.4.patch, HIVE-16295.5.patch,
> HIVE-16295.6.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: (was: HIVE-16295.5.patch)
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch,
> HIVE-16295.3.WIP.patch, HIVE-16295.4.patch, HIVE-16295.5.patch,
> HIVE-16295.6.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: HIVE-16295.5.patch
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch,
> HIVE-16295.3.WIP.patch, HIVE-16295.4.patch, HIVE-16295.5.patch,
> HIVE-16295.5.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495793#comment-16495793 ] Matt McCline commented on HIVE-19118: - The branch-3 Hive QA build disappeared. Resubmit. > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch, HIVE-19118.04.patch, HIVE-19118.05.patch, > HIVE-19118.06-branch-3.patch, HIVE-19118.06.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19118: Status: Patch Available (was: In Progress) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch, HIVE-19118.04.patch, HIVE-19118.05.patch, > HIVE-19118.06-branch-3.patch, HIVE-19118.06.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-19118: --- Assignee: Matt McCline (was: Haifeng Chen) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch, HIVE-19118.04.patch, HIVE-19118.05.patch, > HIVE-19118.06-branch-3.patch, HIVE-19118.06.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-19118: --- Assignee: Haifeng Chen (was: Matt McCline) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch, HIVE-19118.04.patch, HIVE-19118.05.patch, > HIVE-19118.06-branch-3.patch, HIVE-19118.06.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19118: Status: In Progress (was: Patch Available) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch, HIVE-19118.04.patch, HIVE-19118.05.patch, > HIVE-19118.06-branch-3.patch, HIVE-19118.06.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Attachment: HIVE-19529.05.patch > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.04.patch, HIVE-19529.05.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Status: Patch Available (was: In Progress) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.04.patch, HIVE-19529.05.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19529: Status: In Progress (was: Patch Available) > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.04.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[ https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495788#comment-16495788 ] Matt McCline commented on HIVE-19529: - Resubmit. > Vectorization: Date/Timestamp NULL issues > - > > Key: HIVE-19529 > URL: https://issues.apache.org/jira/browse/HIVE-19529 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-19529.04.patch > > > Wrong results found for: > date_add/date_sub > UT areas: > date_add/date_sub > datediff > to_date > interval_year_month + interval_year_month > interval_day_time + interval_day_time > interval_day_time + timestamp > timestamp + interval_day_time > date + interval_day_time > interval_day_time + date > interval_year_month + date > date + interval_year_month > interval_year_month + interval_year_month > timestamp + interval_year_month > date - date > interval_year_month - interval_year_month > interval_day_time - interval_day_time > timestamp - interval_day_time > timestamp - timestamp > date - timestamp > timestamp - date > date - interval_day_time > date - interval_year_month > timestamp - interval_year_month -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19743) hive is not pushing predicate down to HBaseStorageHandler if hive key mapped with hbase is stored as varchar
[
https://issues.apache.org/jira/browse/HIVE-19743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495741#comment-16495741
]
Rajkumar Singh commented on HIVE-19743:
---
this seems to be an intentional change as part of
https://issues.apache.org/jira/browse/HIVE-13831, should not be treated varchar
equivalent to string?
> hive is not pushing predicate down to HBaseStorageHandler if hive key mapped
> with hbase is stored as varchar
>
>
> Key: HIVE-19743
> URL: https://issues.apache.org/jira/browse/HIVE-19743
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive
>Affects Versions: 2.1.0
> Environment: java8,centos7
>Reporter: Rajkumar Singh
>Priority: Major
>
> Steps to Reproduce:
> {code}
> //hbase table
> create 'mytable', 'cf'
> put 'mytable', 'ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4',
> 'cf:message', 'hello world'
> put 'mytable', 'ABCDEF1|GHIJK1|ijj123kl-mn4o-4pq5-678r-st90123u0v41',
> 'cf:foo', 0x0
> // hive table with key stored as varchar
> show create table hbase_table_4;
> +---+--+
> | createtab_stmt |
> +---+--+
> | CREATE EXTERNAL TABLE `hbase_table_4`( |
> | `hbase_key` varchar(80) COMMENT 'from deserializer', |
> | `value` string COMMENT 'from deserializer', |
> | `value1` string COMMENT 'from deserializer') |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.hbase.HBaseSerDe' |
> | STORED BY |
> | 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' |
> | WITH SERDEPROPERTIES ( |
> | 'hbase.columns.mapping'=':key,cf:foo,cf:message', |
> | 'serialization.format'='1') |
> | TBLPROPERTIES ( |
> | 'COLUMN_STATS_ACCURATE'='\{\"BASIC_STATS\":\"true\"}', |
> | 'hbase.table.name'='mytable', |
> | 'numFiles'='0', |
> | 'numRows'='0', |
> | 'rawDataSize'='0', |
> | 'totalSize'='0', |
> | 'transient_lastDdlTime'='1527708430') |
> +---+--+
>
> // hive table key stored as string
> CREATE EXTERNAL TABLE `hbase_table_5`( |
> | `hbase_key` string COMMENT 'from deserializer', |
> | `value` string COMMENT 'from deserializer', |
> | `value1` string COMMENT 'from deserializer') |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.hbase.HBaseSerDe' |
> | STORED BY |
> | 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' |
> | WITH SERDEPROPERTIES ( |
> | 'hbase.columns.mapping'=':key,cf:foo,cf:message', |
> | 'serialization.format'='1') |
> | TBLPROPERTIES ( |
> | 'COLUMN_STATS_ACCURATE'='\{\"BASIC_STATS\":\"true\"}', |
> | 'hbase.table.name'='mytable', |
> | 'numFiles'='0', |
> | 'numRows'='0', |
> | 'rawDataSize'='0', |
> | 'totalSize'='0', |
> | 'transient_lastDdlTime'='1527708520') |
>
> Explain Plan
> explain select * from hbase_table_4 where
> hbase_key='ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4'
> Stage-0
> |
> | Fetch Operator
> |
> | limit:-1
> |
> | Select Operator [SEL_2]
> |
> | Output:["_col0","_col1","_col2"]
> |
> | Filter Operator [FIL_4]
> |
> | predicate:(UDFToString(hbase_key) =
> 'ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4') |
> | TableScan [TS_0]
[jira] [Commented] (HIVE-19743) hive is not pushing predicate down to HBaseStorageHandler if hive key mapped with hbase is stored as varchar
[
https://issues.apache.org/jira/browse/HIVE-19743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495740#comment-16495740
]
Gopal V commented on HIVE-19743:
UDFToString(col) =
instead of
col = cast( as varchar(n))
> hive is not pushing predicate down to HBaseStorageHandler if hive key mapped
> with hbase is stored as varchar
>
>
> Key: HIVE-19743
> URL: https://issues.apache.org/jira/browse/HIVE-19743
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive
>Affects Versions: 2.1.0
> Environment: java8,centos7
>Reporter: Rajkumar Singh
>Priority: Major
>
> Steps to Reproduce:
> {code}
> //hbase table
> create 'mytable', 'cf'
> put 'mytable', 'ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4',
> 'cf:message', 'hello world'
> put 'mytable', 'ABCDEF1|GHIJK1|ijj123kl-mn4o-4pq5-678r-st90123u0v41',
> 'cf:foo', 0x0
> // hive table with key stored as varchar
> show create table hbase_table_4;
> +---+--+
> | createtab_stmt |
> +---+--+
> | CREATE EXTERNAL TABLE `hbase_table_4`( |
> | `hbase_key` varchar(80) COMMENT 'from deserializer', |
> | `value` string COMMENT 'from deserializer', |
> | `value1` string COMMENT 'from deserializer') |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.hbase.HBaseSerDe' |
> | STORED BY |
> | 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' |
> | WITH SERDEPROPERTIES ( |
> | 'hbase.columns.mapping'=':key,cf:foo,cf:message', |
> | 'serialization.format'='1') |
> | TBLPROPERTIES ( |
> | 'COLUMN_STATS_ACCURATE'='\{\"BASIC_STATS\":\"true\"}', |
> | 'hbase.table.name'='mytable', |
> | 'numFiles'='0', |
> | 'numRows'='0', |
> | 'rawDataSize'='0', |
> | 'totalSize'='0', |
> | 'transient_lastDdlTime'='1527708430') |
> +---+--+
>
> // hive table key stored as string
> CREATE EXTERNAL TABLE `hbase_table_5`( |
> | `hbase_key` string COMMENT 'from deserializer', |
> | `value` string COMMENT 'from deserializer', |
> | `value1` string COMMENT 'from deserializer') |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.hbase.HBaseSerDe' |
> | STORED BY |
> | 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' |
> | WITH SERDEPROPERTIES ( |
> | 'hbase.columns.mapping'=':key,cf:foo,cf:message', |
> | 'serialization.format'='1') |
> | TBLPROPERTIES ( |
> | 'COLUMN_STATS_ACCURATE'='\{\"BASIC_STATS\":\"true\"}', |
> | 'hbase.table.name'='mytable', |
> | 'numFiles'='0', |
> | 'numRows'='0', |
> | 'rawDataSize'='0', |
> | 'totalSize'='0', |
> | 'transient_lastDdlTime'='1527708520') |
>
> Explain Plan
> explain select * from hbase_table_4 where
> hbase_key='ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4'
> Stage-0
> |
> | Fetch Operator
> |
> | limit:-1
> |
> | Select Operator [SEL_2]
> |
> | Output:["_col0","_col1","_col2"]
> |
> | Filter Operator [FIL_4]
> |
> | predicate:(UDFToString(hbase_key) =
> 'ABCDEF|GHIJK|ijj123kl-mn4o-4pq5-678r-st90123u0v4') |
> | TableScan [TS_0]
> |
> | Output:["hbase_key","value","value1"]
>
> explain on table
[jira] [Updated] (HIVE-12192) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-12192:
---
Attachment: HIVE-12192.05.patch
> Hive should carry out timestamp computations in UTC
> ---
>
> Key: HIVE-12192
> URL: https://issues.apache.org/jira/browse/HIVE-12192
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Ryan Blue
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Labels: timestamp
> Attachments: HIVE-12192.01.patch, HIVE-12192.02.patch,
> HIVE-12192.03.patch, HIVE-12192.04.patch, HIVE-12192.05.patch,
> HIVE-12192.patch
>
>
> Hive currently uses the "local" time of a java.sql.Timestamp to represent the
> SQL data type TIMESTAMP WITHOUT TIME ZONE. The purpose is to be able to use
> {{Timestamp#getYear()}} and similar methods to implement SQL functions like
> {{year}}.
> When the SQL session's time zone is a DST zone, such as America/Los_Angeles
> that alternates between PST and PDT, there are times that cannot be
> represented because the effective zone skips them.
> {code}
> hive> select TIMESTAMP '2015-03-08 02:10:00.101';
> 2015-03-08 03:10:00.101
> {code}
> Using UTC instead of the SQL session time zone as the underlying zone for a
> java.sql.Timestamp avoids this bug, while still returning correct values for
> {{getYear}} etc. Using UTC as the convenience representation (timestamp
> without time zone has no real zone) would make timestamp calculations more
> consistent and avoid similar problems in the future.
> Notably, this would break the {{unix_timestamp}} UDF that specifies the
> result is with respect to ["the default timezone and default
> locale"|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
> That function would need to be updated to use the
> {{System.getProperty("user.timezone")}} zone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[
https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495730#comment-16495730
]
Hive QA commented on HIVE-19725:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925493/HIVE-19725.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14420 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testShouldDumpMetaDataForNonNativeTableIfSetMeataDataOnly
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925493 - PreCommit-HIVE-Build
> Add ability to dump non-native tables in replication metadata dump
> --
>
> Key: HIVE-19725
> URL: https://issues.apache.org/jira/browse/HIVE-19725
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: Repl, pull-request-available
> Fix For: 3.1.0, 3.0.1, 4.0.0
>
> Attachments: HIVE-19725.01.patch
>
>
> if hive.repl.dump.metadata.only is set to true, allow dumping non native
> tables also.
> Data dump for non-native tables should never be allowed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19741) Update documentation to reflect list of reserved words
[ https://issues.apache.org/jira/browse/HIVE-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495727#comment-16495727 ] Lefty Leverenz commented on HIVE-19741: --- The keyword APPLICATION was added by HIVE-18004 in release 3.0.0. It's a reserved word (since IdentifiersParser.g doesn't list it as nonreserved). > Update documentation to reflect list of reserved words > -- > > Key: HIVE-19741 > URL: https://issues.apache.org/jira/browse/HIVE-19741 > Project: Hive > Issue Type: Improvement > Components: Documentation >Reporter: Matt Burgess >Priority: Minor > > The current list of non-reserved and reserved keywords is on the Hive wiki: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Keywords,Non-reservedKeywordsandReservedKeywords > However it does not match the list in code (see the lexer rules here): > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > On particular example is the "application" keyword, which was discovered > while trying to create a table with a column named "application". > This Jira proposes to align the documentation with the current set of > non-reserved and reserved keywords. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495686#comment-16495686 ] Ashutosh Chauhan commented on HIVE-19690: - +1 pending tests. > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495688#comment-16495688 ] Sergey Shelukhin commented on HIVE-19690: - This problem for me happens both with and without CBO, vectorization, or Tez. It's a fundamental issue with the plan as far as I see... but I'm not sure it would be related to the other issue. The problem is that two GBY operators expect rows with different layout, one pre-processed for distinct by ReduceSinkOperator, one not. However, if there's only one RSOp, it always pre-processes the rows. Not sure why that would affect operator names. > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19725) Add ability to dump non-native tables in replication metadata dump
[
https://issues.apache.org/jira/browse/HIVE-19725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495676#comment-16495676
]
Hive QA commented on HIVE-19725:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11366/dev-support/hive-personality.sh
|
| git revision | master / 53fda73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11366/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add ability to dump non-native tables in replication metadata dump
> --
>
> Key: HIVE-19725
> URL: https://issues.apache.org/jira/browse/HIVE-19725
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: Repl, pull-request-available
> Fix For: 3.1.0, 3.0.1, 4.0.0
>
> Attachments: HIVE-19725.01.patch
>
>
> if hive.repl.dump.metadata.only is set to true, allow dumping non native
> tables also.
> Data dump for non-native tables should never be allowed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19726) ORC date PPD is broken
[ https://issues.apache.org/jira/browse/HIVE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19726: - Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 2.4.0 Status: Resolved (was: Patch Available) Committed to branch-2, branch-3 and master. Did not commit to branch-3.0 as new patch release (3.0.1) would require a new storage-api release. > ORC date PPD is broken > -- > > Key: HIVE-19726 > URL: https://issues.apache.org/jira/browse/HIVE-19726 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0, 3.1.0, 3.0.1, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 2.4.0, 3.1.0, 4.0.0 > > Attachments: HIVE-19726.1.patch > > > When kryo was in version 2.22 we added a fix in HIVE-7222 and later in > HIVE-10819. Now that we have updated kryo to 3.0.3 that old workaround fix > was never removed. The issue was that kryo serialized Timestamp to Date type. > So to recover the timestamp, during deserialization we deserialized *any* > date instance to Timestamp object which is wrong (we don't know if date was > serialized as date or timestamp serialized as date in first place). This > breaks PPD on date time as kryo deserialization always converts Date to > Timestamp breaking PPD because of type mismatch. > Now that we have newer kryo version we can remove the code added in > HIVE-10819. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[
https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495664#comment-16495664
]
Igor Kryvenko commented on HIVE-19690:
--
[~sershe] Hi. Did you face this problem on CBO?
I'm working on [HIVE-12342|https://issues.apache.org/jira/browse/HIVE-12342],
and after setting {{hive.optimize.index.filter}} to true, I faced that my q
tests particularly {{TestTezPerfCliDriver}} have different outputs each launch.
{code:java}
-
Conds:RS_202._col0=RS_186._col0(Inner),Output:["_col1"]
+
Conds:RS_202._col0=RS_184._col0(Inner),Output:["_col1"]
<-Map 12 [SIMPLE_EDGE] vectorized
-PARTITION_ONLY_SHUFFLE [RS_186]
+PARTITION_ONLY_SHUFFLE [RS_184]
PartitionCols:_col0
- Select Operator [SEL_185] (rows=4058
width=1119)
+ Select Operator [SEL_183] (rows=4058
width=1119)
Output:["_col0"]
-Filter Operator [FIL_184] (rows=4058
width=1119)
+Filter Operator [FIL_182] (rows=4058
width=1119)
{code}
I think it is related to this JIRA. What do you think about it?
Thanks, Igor.
> multi-insert query with multiple GBY, and distinct in only some branches can
> produce incorrect results
> --
>
> Key: HIVE-19690
> URL: https://issues.apache.org/jira/browse/HIVE-19690
> Project: Hive
> Issue Type: Bug
>Reporter: Riju Trivedi
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch,
> HIVE-19690.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.
[
https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495644#comment-16495644
]
Hive QA commented on HIVE-19499:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925516/HIVE-19499.02-branch-3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14356 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=253)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_2]
(batchId=162)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail]
(batchId=95)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=304)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11365/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11365/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11365/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925516 - PreCommit-HIVE-Build
> Bootstrap REPL LOAD shall add tasks to create checkpoints for
> db/tables/partitions.
> ---
>
> Key: HIVE-19499
> URL: https://issues.apache.org/jira/browse/HIVE-19499
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-19499.01-branch-3.patch, HIVE-19499.01.patch,
> HIVE-19499.02-branch-3.patch, HIVE-19499.02.patch
>
>
> Currently. bootstrap REPL LOAD expect the target database to be empty or not
> exist to start bootstrap load.
> But, this adds overhead when there is a failure in between bootstrap load and
> there is no way to resume it from where it fails. So, it is needed to create
> checkpoints in table/partitions to skip the completely loaded objects.
> Use the fully qualified path of the dump directory as a checkpoint
> identifier. This should be added to the table / partition properties in hive
> via a task, as the last task in the DAG for table / partition creation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19726) ORC date PPD is broken
[ https://issues.apache.org/jira/browse/HIVE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495639#comment-16495639 ] Prasanth Jayachandran commented on HIVE-19726: -- This fix to storage-api is required for ORC-370. Will commit this patch. After this storage-api has to be released, which ORC-370 has to be use. > ORC date PPD is broken > -- > > Key: HIVE-19726 > URL: https://issues.apache.org/jira/browse/HIVE-19726 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0, 3.1.0, 3.0.1, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19726.1.patch > > > When kryo was in version 2.22 we added a fix in HIVE-7222 and later in > HIVE-10819. Now that we have updated kryo to 3.0.3 that old workaround fix > was never removed. The issue was that kryo serialized Timestamp to Date type. > So to recover the timestamp, during deserialization we deserialized *any* > date instance to Timestamp object which is wrong (we don't know if date was > serialized as date or timestamp serialized as date in first place). This > breaks PPD on date time as kryo deserialization always converts Date to > Timestamp breaking PPD because of type mismatch. > Now that we have newer kryo version we can remove the code added in > HIVE-10819. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19432) HIVE-7575: GetTablesOperation is too slow if the hive has too many databases and tables
[
https://issues.apache.org/jira/browse/HIVE-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495626#comment-16495626
]
Ashutosh Chauhan commented on HIVE-19432:
-
+!
> HIVE-7575: GetTablesOperation is too slow if the hive has too many databases
> and tables
> ---
>
> Key: HIVE-19432
> URL: https://issues.apache.org/jira/browse/HIVE-19432
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 2.2.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19432.01.patch, HIVE-19432.01.patch,
> HIVE-19432.patch
>
>
> GetTablesOperation is too slow since it does not check for the authorization
> for databases and try pulling all the tables from all the databases using
> getTableMeta. for operation like follows
> {code}
> con.getMetaData().getTables("", "", "%", new String[] \{ "TABLE", "VIEW" });
> {code}
> build the getTableMeta call with wildcard *
> {code}
> metastore.HiveMetaStore: 8: get_table_metas : db=* tbl=*
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495622#comment-16495622 ] Sergey Shelukhin commented on HIVE-19690: - Hmm.. I did? Just uploaded again > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19690) multi-insert query with multiple GBY, and distinct in only some branches can produce incorrect results
[ https://issues.apache.org/jira/browse/HIVE-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495618#comment-16495618 ] Ashutosh Chauhan commented on HIVE-19690: - [~sershe] Can you update the RB with latest patch? > multi-insert query with multiple GBY, and distinct in only some branches can > produce incorrect results > -- > > Key: HIVE-19690 > URL: https://issues.apache.org/jira/browse/HIVE-19690 > Project: Hive > Issue Type: Bug >Reporter: Riju Trivedi >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19690.01.patch, HIVE-19690.02.patch, > HIVE-19690.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19462: Status: Patch Available (was: Open) > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.2.patch, HIVE-19462.3.patch, > HIVE-19462.4.patch, HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19462: Status: Open (was: Patch Available) > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.2.patch, HIVE-19462.3.patch, > HIVE-19462.4.patch, HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19462: Attachment: HIVE-19462.4.patch > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.2.patch, HIVE-19462.3.patch, > HIVE-19462.4.patch, HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19237) Only use an operatorId once in a plan
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19237: Attachment: HIVE-19237.08.patch > Only use an operatorId once in a plan > - > > Key: HIVE-19237 > URL: https://issues.apache.org/jira/browse/HIVE-19237 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19237.01.patch, HIVE-19237.02.patch, > HIVE-19237.03.patch, HIVE-19237.04.patch, HIVE-19237.05.patch, > HIVE-19237.05.patch, HIVE-19237.06.patch, HIVE-19237.07.patch, > HIVE-19237.08.patch, HIVE-19237.08.patch > > > Column stats autogather plan part is added from a plan compiled by the driver > itself; however that driver starts to use operatorIds from 1 ; so it's > possible that 2 SEL_1 operators end up in the same plan... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19237) Only use an operatorId once in a plan
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19237: Attachment: HIVE-19237.08.patch > Only use an operatorId once in a plan > - > > Key: HIVE-19237 > URL: https://issues.apache.org/jira/browse/HIVE-19237 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19237.01.patch, HIVE-19237.02.patch, > HIVE-19237.03.patch, HIVE-19237.04.patch, HIVE-19237.05.patch, > HIVE-19237.05.patch, HIVE-19237.06.patch, HIVE-19237.07.patch, > HIVE-19237.08.patch > > > Column stats autogather plan part is added from a plan compiled by the driver > itself; however that driver starts to use operatorIds from 1 ; so it's > possible that 2 SEL_1 operators end up in the same plan... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19499) Bootstrap REPL LOAD shall add tasks to create checkpoints for db/tables/partitions.
[
https://issues.apache.org/jira/browse/HIVE-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495582#comment-16495582
]
Hive QA commented on HIVE-19499:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-11365/patches/PreCommit-HIVE-Build-11365.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bootstrap REPL LOAD shall add tasks to create checkpoints for
> db/tables/partitions.
> ---
>
> Key: HIVE-19499
> URL: https://issues.apache.org/jira/browse/HIVE-19499
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Fix For: 4.0.0
>
> Attachments: HIVE-19499.01-branch-3.patch, HIVE-19499.01.patch,
> HIVE-19499.02-branch-3.patch, HIVE-19499.02.patch
>
>
> Currently. bootstrap REPL LOAD expect the target database to be empty or not
> exist to start bootstrap load.
> But, this adds overhead when there is a failure in between bootstrap load and
> there is no way to resume it from where it fails. So, it is needed to create
> checkpoints in table/partitions to skip the completely loaded objects.
> Use the fully qualified path of the dump directory as a checkpoint
> identifier. This should be added to the table / partition properties in hive
> via a task, as the last task in the DAG for table / partition creation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18684) Race condition in RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18684:
Attachment: HIVE-18684.2.patch
> Race condition in RemoteSparkJobMonitor
> ---
>
> Key: HIVE-18684
> URL: https://issues.apache.org/jira/browse/HIVE-18684
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18684.1.patch, HIVE-18684.2.patch
>
>
> There is a race condition in {{RemoteSparkJobMonitor}}. Sometimes the info in
> {{RemoteSparkJobMonitor#startMonitor.STARTED}} gets printed out, sometimes it
> doesn't. This can be easily verified by running a qtest on
> {{TestMiniSparkOnYarnCliDriver}} and counting the number of times {{Query
> Hive on Spark job}} is printed vs. the number of times {{Finished
> successfully in}} gets printed.
> The issue is that {{RemoteSparkJobMonitor}} runs every one second, and checks
> the state of {{JobHandle}}. Depending on the state, it prints out some
> logging info. The content of the logs contain an implicit assumption that
> logs in the {{STARTED}} state are printed before the logs in the
> {{SUCCEEDED}} state. However, this isn't always the case. The state
> transitions are driven by how long the remote Spark job takes to run, and it
> it finishes within one second then the logs in the {{STARTED}} state never
> printed.
> This can be confusing to users, and there is key debugging information that
> is printed in the {{STARTED}} state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[
https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495579#comment-16495579
]
Hive QA commented on HIVE-19529:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12925691/HIVE-19529.04.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 14426 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11364/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11364/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11364/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12925691 - PreCommit-HIVE-Build
> Vectorization: Date/Timestamp NULL issues
> -
>
> Key: HIVE-19529
> URL: https://issues.apache.org/jira/browse/HIVE-19529
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-19529.04.patch
>
>
> Wrong results found for:
> date_add/date_sub
> UT areas:
> date_add/date_sub
> datediff
> to_date
> interval_year_month + interval_year_month
> interval_day_time + interval_day_time
> interval_day_time + timestamp
> timestamp + interval_day_time
> date + interval_day_time
> interval_day_time + date
> interval_year_month + date
> date + interval_year_month
> interval_year_month + interval_year_month
> timestamp + interval_year_month
> date - date
> interval_year_month - interval_year_month
> interval_day_time - interval_day_time
> timestamp - interval_day_time
> timestamp - timestamp
> date - timestamp
> timestamp - date
> date - interval_day_time
> date - interval_year_month
> timestamp - interval_year_month
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19726) ORC date PPD is broken
[ https://issues.apache.org/jira/browse/HIVE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495566#comment-16495566 ] Prasanth Jayachandran commented on HIVE-19726: -- Actually this patch might alone be not sufficient.. The same tests fail when running on a different time zone (same test failed with -Duser.timezone="Europe/Paris"). The reason is ORC PPD evaluation performs Date.compareTo(Date) which compares millis offset. Date.compareTo(Date) returned 0 in Los Angeles timezone whereas Date.compareTo(Date) return -1 in Paris timezone for the same test case. The fix for this would be in orc side where PPD evaluation has to happen on DateWritables (instead of Date), as DateWritables compare days offset. DateWritable.dateToDays is used for bloomfilter and date column vectors are populated using DateWritable.getDays(). I will create ORC issue for the same. > ORC date PPD is broken > -- > > Key: HIVE-19726 > URL: https://issues.apache.org/jira/browse/HIVE-19726 > Project: Hive > Issue Type: Bug >Affects Versions: 2.4.0, 3.1.0, 3.0.1, 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19726.1.patch > > > When kryo was in version 2.22 we added a fix in HIVE-7222 and later in > HIVE-10819. Now that we have updated kryo to 3.0.3 that old workaround fix > was never removed. The issue was that kryo serialized Timestamp to Date type. > So to recover the timestamp, during deserialization we deserialized *any* > date instance to Timestamp object which is wrong (we don't know if date was > serialized as date or timestamp serialized as date in first place). This > breaks PPD on date time as kryo deserialization always converts Date to > Timestamp breaking PPD because of type mismatch. > Now that we have newer kryo version we can remove the code added in > HIVE-10819. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19662) Upgrade Avro to 1.8.2
[ https://issues.apache.org/jira/browse/HIVE-19662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-19662: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~janulatha] for the work. > Upgrade Avro to 1.8.2 > - > > Key: HIVE-19662 > URL: https://issues.apache.org/jira/browse/HIVE-19662 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19662.1.patch, HIVE-19662.1.patch, > HIVE-19662.1.patch > > > Upgrade Avro to 1.8.2 and use the version from root pom where Avro is > referenced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19493) VectorUDFDateDiffColCol copySelected does not handle nulls correctly
[
https://issues.apache.org/jira/browse/HIVE-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495561#comment-16495561
]
Vihang Karajgaonkar commented on HIVE-19493:
HIVE-19529 introduces a test case which is fixed in this patch. [~mmccline] Do
you think I should wait for HIVE-19529 to merged? or this patch in the current
state can be merged (pending precommit)?
> VectorUDFDateDiffColCol copySelected does not handle nulls correctly
>
>
> Key: HIVE-19493
> URL: https://issues.apache.org/jira/browse/HIVE-19493
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-19493.01.patch, HIVE-19493.02.patch
>
>
> The {{copySelected}} method in {{VectorUDFDateDiffColCol}} class was missed
> during HIVE-18622
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495560#comment-16495560 ] Steve Yeom commented on HIVE-19417: --- 1. MetaStore Schema Changes for the patch 04 1.1 New table UPD_TXNS A record is created per transaction per table. - UPDATE_ID: primary key column, generated by Datanucleus when specified at "package.jdo" as datastore-identity - TBL_ID: TBLS.TBL_ID referencing column. A foreign key is created for this column referencing TBLS.TBL_ID. - STATE: this is deleted for the next patch version. - TXN_ID: Transaction id of the transaction to insert the row. - WRITEID_LIST: writeIdList for the table of the transaction 1.2 Modification for TBLS and PARTITIONS tables - A new column TXN_ID: transaction id of the UPD_TXNS. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch, HIVE-19417.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495556#comment-16495556 ] Eugene Koifman commented on HIVE-19598: --- 11380 in build queue > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-19598.02.patch, HIVE-19598.05.patch, > HIVE-19598.06.patch > > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19529) Vectorization: Date/Timestamp NULL issues
[
https://issues.apache.org/jira/browse/HIVE-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495551#comment-16495551
]
Hive QA commented on HIVE-19529:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2333 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
47s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} ql: The patch generated 224 new + 3292 unchanged - 5
fixed = 3516 total (was 3297) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
47s{color} | {color:red} ql generated 8 new + 2270 unchanged - 63 fixed = 2278
total (was 2333) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnBetween.left At
FilterCharColumnBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnBetween.left At
FilterCharColumnBetween.java:[line 44] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnBetween.right At
FilterCharColumnBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnBetween.right At
FilterCharColumnBetween.java:[line 45] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnNotBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnNotBetween.left At
FilterCharColumnNotBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnNotBetween.left At
FilterCharColumnNotBetween.java:[line 44] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterCharColumnNotBetween(int,
byte[], byte[]) may expose internal representation by storing an externally
mutable object into FilterCharColumnNotBetween.right At
FilterCharColumnNotBetween.java:expose internal representation by storing an
externally mutable object into FilterCharColumnNotBetween.right At
FilterCharColumnNotBetween.java:[line 45] |
| | new
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterVarCharColumnBetween(int,
byte[], byt
[jira] [Updated] (HIVE-19675) Cast to timestamps on Druid time column leads to an exception
[
https://issues.apache.org/jira/browse/HIVE-19675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-19675:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Slim
[~vgarg] Can we get this in branch-3 as well. Since change is only in
druid-handler/ it has low risk to break any other tests. So, doesn't warrant
another run for branch-3.
> Cast to timestamps on Druid time column leads to an exception
> -
>
> Key: HIVE-19675
> URL: https://issues.apache.org/jira/browse/HIVE-19675
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19675.2.patch, HIVE-19675.3.patch,
> HIVE-19675.4.patch, HIVE-19675.5.patch, HIVE-19675.patch
>
>
> The following query fail due to a formatting issue.
> {code}
> SELECT CAST(`ssb_druid_100`.`__time` AS TIMESTAMP) AS `x_time`,
> . . . . . . . . . . . . . . . .> SUM(`ssb_druid_100`.`lo_revenue`) AS
> `sum_lo_revenue_ok`
> . . . . . . . . . . . . . . . .> FROM `druid_ssb`.`ssb_druid_100`
> `ssb_druid_100`
> . . . . . . . . . . . . . . . .> GROUP BY CAST(`ssb_druid_100`.`__time` AS
> TIMESTAMP);
> {code}
> Exception
> {code}
> Error: java.io.IOException: java.lang.NumberFormatException: For input
> string: "1991-12-31 19:00:00" (state=,code=0)
> {code}
> [~jcamachorodriguez] maybe this is fixed by your upcoming patches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
