[jira] [Commented] (HIVE-21247) Webhcat beeline in secure mode
[
https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778998#comment-16778998
]
Hive QA commented on HIVE-21247:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958323/HIVE-21247.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath (batchId=205)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16262/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16262/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16262/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958323 - PreCommit-HIVE-Build
> Webhcat beeline in secure mode
> --
>
> Key: HIVE-21247
> URL: https://issues.apache.org/jira/browse/HIVE-21247
> Project: Hive
> Issue Type: Improvement
> Components: WebHCat
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21247.1.patch
>
>
> Follow up HIVE-20550, we need to make beeline work in secure mode. That
> means, we need to get a delegation token from hiveserver2, and pass that to
> beeline. This is similar to HIVE-5133, I make two changes:
> 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection,
> and pass along
> 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION,
> and extract delegation token if exists
> There's one config needed in webhcat-site.xml, to point
> "templeton.hive.hs2.url" to hive jdbc url.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21247) Webhcat beeline in secure mode
[
https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778996#comment-16778996
]
Hive QA commented on HIVE-21247:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} beeline in master has 45 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
7s{color} | {color:red} packaging in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} jdbc: The patch generated 1 new + 33 unchanged - 0
fixed = 34 total (was 33) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/webhcat/svr: The patch generated 1 new + 74
unchanged - 0 fixed = 75 total (was 74) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
12s{color} | {color:red} root: The patch generated 2 new + 153 unchanged - 0
fixed = 155 total (was 153) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} hcatalog/webhcat/svr generated 1 new + 96 unchanged -
0 fixed = 97 total (was 96) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hcatalog/webhcat/svr |
| | Dead store to c in
org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.buildHS2DelegationToken(String)
At
TempletonControllerJob.java:org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.buildHS2DelegationToken(String)
At TempletonControllerJob.java:[line 210] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16262/dev-support/hive-personality.sh
|
| git revision | master / 77b9c3c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16262/yetus/patch-mvninstall-packaging.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16262/yetus/diff-checkstyle-jdbc.txt
|
| checkstyle |
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778986#comment-16778986 ] Ashutosh Chauhan commented on HIVE-20801: - hive.txn.strict.locking.mode=false should be sufficient for that. No? > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778985#comment-16778985 ] Gopal V commented on HIVE-20801: bq. Is that acquiring locks for is slow? No, "insert into" locks out reads for non-ACID tables for atomicity. > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778983#comment-16778983
]
Gopal V commented on HIVE-21225:
The specific improvement I'm looking at is to make the
{{List childrenWithId}} in getAcidState to be a recursive
listing holder, which is a local change to a single function.
Eugene's suggestion is a bit better for skipping one of the looks ups, but the
crucial one I'm looking at is to have that dir listing be all-depth instead of
single level (FYI, all-depth listings are faster in S3 than one level deep,
since the recursive one is just a prefix-lookup, while the one-level is a
prefix-lookup + discard anything which has another "/" later).
> ACID: getAcidState() should cache a recursive dir listing locally
> -
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be
> answered by making a single recursive listDir call and reusing the same data
> to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed
> directory snapshot instead of interacting with the NameNode or ObjectStore
> within the inner loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778979#comment-16778979 ] Ashutosh Chauhan commented on HIVE-21225: - [~vgumashta] Which approach you are thinking here? [~ekoifman]'s of encoding identifier in file names or [~gopalv]'s single recursive call. If we change names of dirs not sure if that will have any impact on data in existing tables. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778980#comment-16778980 ] Ashutosh Chauhan commented on HIVE-20801: - bq. Just that Hive is slowed down by a magnitude when ACID is enabled for even 1 tables. >From where this slowness coming from? Is that acquiring locks for is slow? > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778977#comment-16778977 ] Na Li commented on HIVE-21320: -- patch 9 is rebased on latest code at https://github.com/apache/hive/commit/77b9c3c79953121c19554a7de6953a6e21de9db6 The failed tests are likely caused by daijy 's commit at https://github.com/apache/hive/commit/3e9614eebd19f9843b28d97aaee1c3dfb815fb3d, which is reverted now. > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch, > HIVE-21320.008.patch, HIVE-21320.008.patch, HIVE-21320.009.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-21320: - Attachment: HIVE-21320.009.patch > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch, > HIVE-21320.008.patch, HIVE-21320.008.patch, HIVE-21320.009.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[
https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778975#comment-16778975
]
Gopal V commented on HIVE-21329:
LGTM - +1
{code}
2019-02-27 01:35:38,588 [INFO] [TezChild]
|writers.UnorderedPartitionedKVWriter|: Reducer_2: numBuffers=2,
sizePerBuffer=524288, skipBuffers=true, numPartitions=1,
availableMemory=1048576, maxSingleBufferSizeBytes=2147483647,
pipelinedShuffle=false, isFinalMergeEnabled=false, numPartitions=1,
reportPartitionStats=ENABLED
2019-02-27 01:35:38,592 [INFO] [TezChild] |tez.MapRecordProcessor|: Main input
name is Map 1
{code}
The allocation does show up in the timings in quick tests.
{code}
0: jdbc:hive2://localhost:10013/> set
hive.tez.unordered.output.buffer.size.mb=1024;
0: jdbc:hive2://localhost:10013/> select count(1) from store_sales order by 1;
INFO : Completed executing
command(queryId=gopal_20190227015023_75eeb0ec-edd2-4248-8122-a3147c0bb502);
Time taken: 5.436 seconds
{code}
with -1
{code}
INFO : Completed executing
command(queryId=gopal_20190227015049_2068d1e9-b9ac-463a-8bdb-738966a480a7);
Time taken: 3.645 seconds
{code}
> Custom Tez runtime unordered output buffer size depending on operator pipeline
> --
>
> Key: HIVE-21329
> URL: https://issues.apache.org/jira/browse/HIVE-21329
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21329.patch
>
>
> For instance, if we have a reduce sink operator with no keys followed by a
> Group By (merge partial), we can decrease the output buffer size since we
> will only produce a single row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20854) Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change to 2
[
https://issues.apache.org/jira/browse/HIVE-20854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778946#comment-16778946
]
Prasanth Jayachandran commented on HIVE-20854:
--
+1
there is another prod outage today related to stale entries in zk for LLAP.
Longer term we could have separate configs for zk lock manager and LLAP.
> Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change
> to 2
> -
>
> Key: HIVE-20854
> URL: https://issues.apache.org/jira/browse/HIVE-20854
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-20854.1.patch
>
>
> {code}
> HIVE_ZOOKEEPER_SESSION_TIMEOUT("hive.zookeeper.session.timeout",
> "120ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "ZooKeeper client's session timeout (in milliseconds). The client is
> disconnected, and as a result, all locks released, \n" +
> "if a heartbeat is not sent in the timeout."),
> {code}
> That's 1,200,000ms which is too long for all practical purposes - a 20 minute
> outage in case a node has a failure is too long.
> That is too long for the JDBC load-balancing, LLAP failure tolerance and the
> lock manager expiry.
> Change to 2 minutes, as a sensible default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18920) CBO: Initialize the Janino providers ahead of 1st query
[ https://issues.apache.org/jira/browse/HIVE-18920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778965#comment-16778965 ] Ashutosh Chauhan commented on HIVE-18920: - then it is no longer an issue? [~jcamachorodriguez] can you please confirm and resolve. > CBO: Initialize the Janino providers ahead of 1st query > --- > > Key: HIVE-18920 > URL: https://issues.apache.org/jira/browse/HIVE-18920 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Hive Calcite metadata providers are compiled when the 1st query comes in. > If a second query arrives before the 1st one has built a metadata provider, > it will also try to do the same thing, because the cache is not populated yet. > With 1024 concurrent users, it takes 6 minutes for the 1st query to finish > fighting all the other queries which are trying to load that cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Attachment: HIVE-21292.15.patch > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch, > HIVE-21292.15.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Status: Patch Available (was: Open) > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch, > HIVE-21292.15.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Status: Open (was: Patch Available) > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21294) Vectorization: 1-reducer Shuffle can skip the object hash functions
[
https://issues.apache.org/jira/browse/HIVE-21294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778949#comment-16778949
]
Gopal V commented on HIVE-21294:
{code}
< Statistics: Num rows: 220 Data size: 41800 Basic
stats: COMPLETE Column stats: COMPLETE
---
> Statistics: Num rows: 150 Data size: 28500 Basic
> stats: COMPLETE Column stats: COMPLETE
{code}
diffs in murmur_hash_migration.q seems to be stats related
> Vectorization: 1-reducer Shuffle can skip the object hash functions
> ---
>
> Key: HIVE-21294
> URL: https://issues.apache.org/jira/browse/HIVE-21294
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21294.2.patch, HIVE-21294.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> VectorReduceSinkObjectHashOperator can skip the object hashing entirely if
> the reducer count = 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21294) Vectorization: 1-reducer Shuffle can skip the object hash functions
[
https://issues.apache.org/jira/browse/HIVE-21294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778948#comment-16778948
]
Hive QA commented on HIVE-21294:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960276/HIVE-21294.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[murmur_hash_migration]
(batchId=175)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16261/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16261/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16261/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960276 - PreCommit-HIVE-Build
> Vectorization: 1-reducer Shuffle can skip the object hash functions
> ---
>
> Key: HIVE-21294
> URL: https://issues.apache.org/jira/browse/HIVE-21294
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21294.2.patch, HIVE-21294.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> VectorReduceSinkObjectHashOperator can skip the object hashing entirely if
> the reducer count = 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18920) CBO: Initialize the Janino providers ahead of 1st query
[ https://issues.apache.org/jira/browse/HIVE-18920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778938#comment-16778938 ] Gopal V commented on HIVE-18920: Last I checked, this accidentally happens when MV registry starts up. > CBO: Initialize the Janino providers ahead of 1st query > --- > > Key: HIVE-18920 > URL: https://issues.apache.org/jira/browse/HIVE-18920 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Hive Calcite metadata providers are compiled when the 1st query comes in. > If a second query arrives before the 1st one has built a metadata provider, > it will also try to do the same thing, because the cache is not populated yet. > With 1024 concurrent users, it takes 6 minutes for the 1st query to finish > fighting all the other queries which are trying to load that cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18920) CBO: Initialize the Janino providers ahead of 1st query
[ https://issues.apache.org/jira/browse/HIVE-18920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778932#comment-16778932 ] Ashutosh Chauhan commented on HIVE-18920: - Can this be done at HS2 process startup time? > CBO: Initialize the Janino providers ahead of 1st query > --- > > Key: HIVE-18920 > URL: https://issues.apache.org/jira/browse/HIVE-18920 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Hive Calcite metadata providers are compiled when the 1st query comes in. > If a second query arrives before the 1st one has built a metadata provider, > it will also try to do the same thing, because the cache is not populated yet. > With 1024 concurrent users, it takes 6 minutes for the 1st query to finish > fighting all the other queries which are trying to load that cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-21320: - Attachment: HIVE-21320.008.patch > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch, > HIVE-21320.008.patch, HIVE-21320.008.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=205002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-205002 ] ASF GitHub Bot logged work on HIVE-21283: - Author: ASF GitHub Bot Created on: 27/Feb/19 05:38 Start Date: 27/Feb/19 05:38 Worklog Time Spent: 10m Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms for the existing functions URL: https://github.com/apache/hive/pull/540#issuecomment-467731239 Hi @sankarh Can you please review the code and merge it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 205002) Time Spent: 1h 20m (was: 1h 10m) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.2.PATCH, HIVE.21283.PATCH > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21294) Vectorization: 1-reducer Shuffle can skip the object hash functions
[
https://issues.apache.org/jira/browse/HIVE-21294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778911#comment-16778911
]
Hive QA commented on HIVE-21294:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 3 unchanged - 2
fixed = 3 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16261/dev-support/hive-personality.sh
|
| git revision | master / 77b9c3c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16261/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: 1-reducer Shuffle can skip the object hash functions
> ---
>
> Key: HIVE-21294
> URL: https://issues.apache.org/jira/browse/HIVE-21294
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal V
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21294.2.patch, HIVE-21294.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> VectorReduceSinkObjectHashOperator can skip the object hashing entirely if
> the reducer count = 1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20057) For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute change not reflecting for non-CAPS
[
https://issues.apache.org/jira/browse/HIVE-20057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778904#comment-16778904
]
Sankar Hariappan commented on HIVE-20057:
-
[~animenon], [~rmsm...@gmail.com]
I think, this issue doesn't exist in master code.
We use Boolean.parseBoolean to read the "EXTERNAL" property which ignores case.
Also, the fix in the patch is to use equalsIgnoreCase when compare the
TableType and not "EXTERNAL" property which is irrelevant.
If I misunderstood your scenario, please let me know.
Also, it would be better to add a unit test to reproduce this bug in master
code.
> For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute
> change not reflecting for non-CAPS
>
>
> Key: HIVE-20057
> URL: https://issues.apache.org/jira/browse/HIVE-20057
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: All Versions
>Reporter: Anirudh
>Assignee: Anirudh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: hive20057.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hive EXTERNAL table shown as MANAGED after conversion using
> {code} ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='True')
> {code}
>
> The DESCRIBE FORMATTED shows:
> {code}
> Table Type: MANAGED_TABLE
> Table Parameters:
> EXTERNAL True
> {code}
>
> This is actually a External table but is shown wrongly, as 'True' was used in
> place of 'TRUE' in the ALTER statement.
> Issue explained here:
> [StakOverflow - Hive Table is MANAGED or
> EXTERNAL|https://stackoverflow.com/questions/51103317/hive-table-is-managed-or-external/51142873#51142873]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[
https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778887#comment-16778887
]
Hive QA commented on HIVE-21320:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960274/HIVE-21320.008.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16260/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16260/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16260/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960274 - PreCommit-HIVE-Build
> get_fields() and get_tables_by_type() are not protected by HMS server access
> control
>
>
> Key: HIVE-21320
> URL: https://issues.apache.org/jira/browse/HIVE-21320
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch,
> HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch,
> HIVE-21320.008.patch
>
>
> User without any privilege can call these functions and get all meta data
> back as if user has full access privilege.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[
https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778862#comment-16778862
]
Hive QA commented on HIVE-21320:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16260/dev-support/hive-personality.sh
|
| git revision | master / 77b9c3c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16260/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> get_fields() and get_tables_by_type() are not protected by HMS server access
> control
>
>
> Key: HIVE-21320
> URL: https://issues.apache.org/jira/browse/HIVE-21320
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch,
> HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch,
> HIVE-21320.008.patch
>
>
> User without any privilege can call these functions and get all meta data
> back as if user has full access privilege.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778846#comment-16778846 ] BELUGA BEHR commented on HIVE-21240: All unit tests are passing [~bslim] [~kgyrtkirk]. Please consider this patch for inclusion into the project. I understand there is some hesitation regarding the change in return type. Previous a native array was returned and now a Collection (List) is returned by the SerDe. I think it's better to work with Java Collections instead of native arrays and if we're going to change the return value at all, this is an appropriate time to introduce such a change, i.e., in a major (4.0) release. > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[
https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778848#comment-16778848
]
Hive QA commented on HIVE-21329:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960269/HIVE-21329.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath (batchId=205)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16259/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16259/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16259/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960269 - PreCommit-HIVE-Build
> Custom Tez runtime unordered output buffer size depending on operator pipeline
> --
>
> Key: HIVE-21329
> URL: https://issues.apache.org/jira/browse/HIVE-21329
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21329.patch
>
>
> For instance, if we have a reduce sink operator with no keys followed by a
> Group By (merge partial), we can decrease the output buffer size since we
> will only produce a single row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778846#comment-16778846 ] BELUGA BEHR edited comment on HIVE-21240 at 2/27/19 3:44 AM: - All unit tests are passing [~bslim] [~kgyrtkirk]. Please consider this patch for inclusion into the project. I understand there is some hesitation regarding the change in return type. Previous a native array was returned and now (with this patch) a Collection (List) is returned by the SerDe. I think it's better to work with Java Collections instead of native arrays and if we're going to change the return value, this is an appropriate time to introduce such a change, i.e., in a major (4.0) release. was (Author: belugabehr): All unit tests are passing [~bslim] [~kgyrtkirk]. Please consider this patch for inclusion into the project. I understand there is some hesitation regarding the change in return type. Previous a native array was returned and now (with this patch) a Collection (List) is returned by the SerDe. I think it's better to work with Java Collections instead of native arrays and if we're going to change the return value at all, this is an appropriate time to introduce such a change, i.e., in a major (4.0) release. > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778846#comment-16778846 ] BELUGA BEHR edited comment on HIVE-21240 at 2/27/19 3:44 AM: - All unit tests are passing [~bslim] [~kgyrtkirk]. Please consider this patch for inclusion into the project. I understand there is some hesitation regarding the change in return type. Previous a native array was returned and now (with this patch) a Collection (List) is returned by the SerDe. I think it's better to work with Java Collections instead of native arrays and if we're going to change the return value at all, this is an appropriate time to introduce such a change, i.e., in a major (4.0) release. was (Author: belugabehr): All unit tests are passing [~bslim] [~kgyrtkirk]. Please consider this patch for inclusion into the project. I understand there is some hesitation regarding the change in return type. Previous a native array was returned and now a Collection (List) is returned by the SerDe. I think it's better to work with Java Collections instead of native arrays and if we're going to change the return value at all, this is an appropriate time to introduce such a change, i.e., in a major (4.0) release. > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reopened HIVE-21247: --- > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778826#comment-16778826 ] Daniel Dai commented on HIVE-21247: --- Missing the ptest. Revert the patch and run ptest first. > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21247: -- Status: Patch Available (was: Reopened) > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[
https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778818#comment-16778818
]
Hive QA commented on HIVE-21329:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 2 new + 428 unchanged - 0
fixed = 430 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16259/dev-support/hive-personality.sh
|
| git revision | master / 3e9614e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16259/yetus/diff-checkstyle-common.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16259/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Custom Tez runtime unordered output buffer size depending on operator pipeline
> --
>
> Key: HIVE-21329
> URL: https://issues.apache.org/jira/browse/HIVE-21329
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21329.patch
>
>
> For instance, if we have a reduce sink operator with no keys followed by a
> Group By (merge partial), we can decrease the output buffer size since we
> will only produce a single row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778809#comment-16778809 ] Na Li commented on HIVE-21320: -- [~pvary] Patch 6 did not trigger test. Patch 7 is the same as patch 6. Its tests failed because it was not rebased on latest code. Patch 8 is uploaded now. The difference between patch 6 and patch 8 + Patch 8 is rebased on lated code + Patch 8 fixed style issues of Patch 6 and 7 > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch, > HIVE-21320.008.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21294) Vectorization: 1-reducer Shuffle can skip the object hash functions
[ https://issues.apache.org/jira/browse/HIVE-21294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-21294: -- Attachment: HIVE-21294.3.patch > Vectorization: 1-reducer Shuffle can skip the object hash functions > --- > > Key: HIVE-21294 > URL: https://issues.apache.org/jira/browse/HIVE-21294 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21294.2.patch, HIVE-21294.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > VectorReduceSinkObjectHashOperator can skip the object hashing entirely if > the reducer count = 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778803#comment-16778803 ] Sankar Hariappan commented on HIVE-21286: - [~maheshk114], Can you please review? > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.rollback.bootstrap.load= > Hive will use this config only if the current dump is bootstrap dump or > combined bootstrap in incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-21320: - Attachment: HIVE-21320.008.patch > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch, > HIVE-21320.008.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[
https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778799#comment-16778799
]
Hive QA commented on HIVE-21320:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960246/HIVE-21320.007.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus (batchId=205)
org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath (batchId=205)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16258/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16258/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16258/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960246 - PreCommit-HIVE-Build
> get_fields() and get_tables_by_type() are not protected by HMS server access
> control
>
>
> Key: HIVE-21320
> URL: https://issues.apache.org/jira/browse/HIVE-21320
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch,
> HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch
>
>
> User without any privilege can call these functions and get all meta data
> back as if user has full access privilege.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21329: --- Attachment: HIVE-21329.patch > Custom Tez runtime unordered output buffer size depending on operator pipeline > -- > > Key: HIVE-21329 > URL: https://issues.apache.org/jira/browse/HIVE-21329 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21329.patch > > > For instance, if we have a reduce sink operator with no keys followed by a > Group By (merge partial), we can decrease the output buffer size since we > will only produce a single row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21329: --- Status: Patch Available (was: In Progress) > Custom Tez runtime unordered output buffer size depending on operator pipeline > -- > > Key: HIVE-21329 > URL: https://issues.apache.org/jira/browse/HIVE-21329 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21329.patch > > > For instance, if we have a reduce sink operator with no keys followed by a > Group By (merge partial), we can decrease the output buffer size since we > will only produce a single row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21329 started by Jesus Camacho Rodriguez. -- > Custom Tez runtime unordered output buffer size depending on operator pipeline > -- > > Key: HIVE-21329 > URL: https://issues.apache.org/jira/browse/HIVE-21329 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21329.patch > > > For instance, if we have a reduce sink operator with no keys followed by a > Group By (merge partial), we can decrease the output buffer size since we > will only produce a single row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21329) Custom Tez runtime unordered output buffer size depending on operator pipeline
[ https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21329: -- > Custom Tez runtime unordered output buffer size depending on operator pipeline > -- > > Key: HIVE-21329 > URL: https://issues.apache.org/jira/browse/HIVE-21329 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > For instance, if we have a reduce sink operator with no keys followed by a > Group By (merge partial), we can decrease the output buffer size since we > will only produce a single row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[
https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778773#comment-16778773
]
Hive QA commented on HIVE-21320:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 413 unchanged - 0 fixed = 418 total (was 413) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16258/dev-support/hive-personality.sh
|
| git revision | master / 3e9614e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16258/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16258/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> get_fields() and get_tables_by_type() are not protected by HMS server access
> control
>
>
> Key: HIVE-21320
> URL: https://issues.apache.org/jira/browse/HIVE-21320
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch,
> HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch
>
>
> User without any privilege can call these functions and get all meta data
> back as if user has full access privilege.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778762#comment-16778762
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960245/HIVE-21240.11.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15829 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16256/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16256/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16256/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960245 - PreCommit-HIVE-Build
> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch,
> HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch,
> HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch,
> HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch,
> HIVE-24240.8.patch, kafka_storage_handler.diff
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-21320: - Status: Patch Available (was: In Progress) > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778738#comment-16778738
]
Hive QA commented on HIVE-21240:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} serde: The patch generated 0 new + 4 unchanged - 25
fixed = 4 total (was 29) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 5
fixed = 6 total (was 11) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} serde generated 0 new + 193 unchanged - 4 fixed =
193 total (was 197) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16256/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16256/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>
[jira] [Commented] (HIVE-21312) FSStatsAggregator::connect is slow
[ https://issues.apache.org/jira/browse/HIVE-21312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778755#comment-16778755 ] Rajesh Balamohan commented on HIVE-21312: - Sure. I will look into this today. > FSStatsAggregator::connect is slow > -- > > Key: HIVE-21312 > URL: https://issues.apache.org/jira/browse/HIVE-21312 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-21312.1.patch, HIVE-21312.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved HIVE-21247. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Patch pushed to master. Thanks Thejas for reviewing! > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20550) Switch WebHCat to use beeline to submit Hive queries
[ https://issues.apache.org/jira/browse/HIVE-20550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20550: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch pushed to master. Thanks Thejas for review! > Switch WebHCat to use beeline to submit Hive queries > > > Key: HIVE-20550 > URL: https://issues.apache.org/jira/browse/HIVE-20550 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20550.1.patch > > > Since hive cli is deprecated, we shall switch WebHCat to use beeline instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778694#comment-16778694
]
Hive QA commented on HIVE-16924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 8 new + 639 unchanged - 13
fixed = 647 total (was 652) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
12s{color} | {color:red} root: The patch generated 8 new + 647 unchanged - 13
fixed = 655 total (was 660) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} ql generated 0 new + 2260 unchanged - 2 fixed = 2260
total (was 2262) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16255/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16255/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16255/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16255/yetus/whitespace-eol.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16255/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch
>
> Time Spent: 3h 10m
> Remaining
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778702#comment-16778702
]
Hive QA commented on HIVE-16924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960241/HIVE-16924.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=92)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16255/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16255/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16255/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960241 - PreCommit-HIVE-Build
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Summary: LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on
right side (HiveJoinAddNotNullRule bails out for outer joins) (was:
HiveJoinAddNotNullRule bails out for outer joins)
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-21323) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
[
https://issues.apache.org/jira/browse/HIVE-21323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg resolved HIVE-21323.
Resolution: Duplicate
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> -
>
> Key: HIVE-21323
> URL: https://issues.apache.org/jira/browse/HIVE-21323
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
>
> {code:sql}
> select a.id from a left outer join c on a.id = c.id
> {code}
> CBO plan:
> {code:sql}
> iveProject(id=[$0])
> HiveJoin(condition=[=($0, $1)], joinType=[left], algorithm=[none],
> cost=[{6.0 rows, 0.0 cpu, 0.0 io}])
> HiveProject(id=[$0])
> HiveTableScan(table=[[hive_21322, a]], table:alias=[a])
> HiveProject(id=[$0])
> HiveTableScan(table=[[hive_21322, c]], table:alias=[c])
> {code}
> Explain Plan:
> {code:sql}
> Stage: Stage-1
> Tez
> DagId: vgarg_20190225222008_083d8041-b5dc-4af1-9dac-4ff5305ab864:10
> Edges:
> Map 1 <- Map 2 (BROADCAST_EDGE)
> DagName: vgarg_20190225222008_083d8041-b5dc-4af1-9dac-4ff5305ab864:10
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: a
> Statistics: Num rows: 3 Data size: 255 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: id (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 3 Data size: 255 Basic stats:
> COMPLETE Column stats: COMPLETE
> Map Join Operator
> condition map:
>Left Outer Join 0 to 1
> keys:
> 0 _col0 (type: string)
> 1 _col0 (type: string)
> outputColumnNames: _col0
> input vertices:
> 1 Map 2
> Statistics: Num rows: 3 Data size: 255 Basic stats:
> COMPLETE Column stats: COMPLETE
> HybridGraceHashJoin: true
> File Output Operator
> compressed: false
> Statistics: Num rows: 3 Data size: 255 Basic stats:
> COMPLETE Column stats: COMPLETE
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Execution mode: vectorized
> Map 2
> Map Operator Tree:
> TableScan
> alias: c
> Statistics: Num rows: 3 Data size: 258 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: id (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 3 Data size: 258 Basic stats:
> COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: string)
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 3 Data size: 258 Basic stats:
> COMPLETE Column stats: COMPLETE
> Execution mode: vectorized
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
> There is no IS NOT NULL filter on {{c.id}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21230) HiveJoinAddNotNullRule bails out for outer joins
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-21230:
--
Assignee: Vineet Garg
> HiveJoinAddNotNullRule bails out for outer joins
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21320) get_fields() and get_tables_by_type() are not protected by HMS server access control
[ https://issues.apache.org/jira/browse/HIVE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated HIVE-21320: - Attachment: HIVE-21320.007.patch > get_fields() and get_tables_by_type() are not protected by HMS server access > control > > > Key: HIVE-21320 > URL: https://issues.apache.org/jira/browse/HIVE-21320 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: HIVE-21320.001.patch, HIVE-21320.005.patch, > HIVE-21320.005.patch, HIVE-21320.006.patch, HIVE-21320.007.patch > > > User without any privilege can call these functions and get all meta data > back as if user has full access privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[
https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778639#comment-16778639
]
Hive QA commented on HIVE-21292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960244/HIVE-21292.14.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16254/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16254/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16254/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-26 22:22:51.440
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16254/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-26 22:22:51.443
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 20abb4e HIVE-21197 : Hive replication can add duplicate data
during migration to a target with hive.strict.managed.tables enabled (Mahesh
Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 20abb4e HIVE-21197 : Hive replication can add duplicate data
during migration to a target with hive.strict.managed.tables enabled (Mahesh
Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-26 22:22:52.568
+ rm -rf ../yetus_PreCommit-HIVE-Build-16254
+ mkdir ../yetus_PreCommit-HIVE-Build-16254
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16254
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16254/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateDatabaseHook.java:
does not exist in index
error:
a/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java:
does not exist in index
error:
a/hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java:
does not exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java: does not
exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadDatabase.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/AlterDatabaseHandler.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/CreateDatabaseHandler.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/DropDatabaseHandler.java:
does not exist in index
error:
[jira] [Commented] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[
https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778637#comment-16778637
]
Hive QA commented on HIVE-21286:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960224/HIVE-21286.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15820 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16253/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16253/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16253/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960224 - PreCommit-HIVE-Build
> Hive should support clean-up of previously bootstrapped tables when retry
> from different dump.
> --
>
> Key: HIVE-21286
> URL: https://issues.apache.org/jira/browse/HIVE-21286
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21286.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If external tables are enabled for replication on an existing repl policy,
> then bootstrapping of external tables are combined with incremental dump.
> If incremental bootstrap load fails with non-retryable error for which user
> will have to manually drop all the external tables before trying with another
> bootstrap dump. For full bootstrap, to retry with different dump, we
> suggested user to drop the DB but in this case they need to manually drop all
> the external tables which is not so user friendly. So, need to handle it in
> Hive side as follows.
> REPL LOAD takes additional config (passed by user in WITH clause) that says,
> drop all the tables which are bootstrapped from previous dump.
> hive.repl.rollback.bootstrap.load=
> Hive will use this config only if the current dump is bootstrap dump or
> combined bootstrap in incremental dump.
> Caution to be taken by user that this config should not be passed if previous
> REPL LOAD (with bootstrap) was successful or any successful incremental
> dump+load happened after "previous_bootstrap_dump_dir".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21247) Webhcat beeline in secure mode
[ https://issues.apache.org/jira/browse/HIVE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778636#comment-16778636 ] Thejas M Nair commented on HIVE-21247: -- +1 > Webhcat beeline in secure mode > -- > > Key: HIVE-21247 > URL: https://issues.apache.org/jira/browse/HIVE-21247 > Project: Hive > Issue Type: Improvement > Components: WebHCat >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21247.1.patch > > > Follow up HIVE-20550, we need to make beeline work in secure mode. That > means, we need to get a delegation token from hiveserver2, and pass that to > beeline. This is similar to HIVE-5133, I make two changes: > 1. Make a jdbc connection to hs2, pull delegation token from HiveConnection, > and pass along > 2. In hive jdbc driver, check for token file in HADOOP_TOKEN_FILE_LOCATION, > and extract delegation token if exists > There's one config needed in webhcat-site.xml, to point > "templeton.hive.hs2.url" to hive jdbc url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20550) Switch WebHCat to use beeline to submit Hive queries
[ https://issues.apache.org/jira/browse/HIVE-20550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778635#comment-16778635 ] Thejas M Nair commented on HIVE-20550: -- +1 > Switch WebHCat to use beeline to submit Hive queries > > > Key: HIVE-20550 > URL: https://issues.apache.org/jira/browse/HIVE-20550 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20550.1.patch > > > Since hive cli is deprecated, we shall switch WebHCat to use beeline instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.11.patch > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.11.patch, HIVE-21240.2.patch, > HIVE-21240.3.patch, HIVE-21240.4.patch, HIVE-21240.5.patch, > HIVE-21240.6.patch, HIVE-21240.7.patch, HIVE-21240.9.patch, > HIVE-24240.8.patch, kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Status: Patch Available (was: Open) > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778612#comment-16778612 ] Vineet Garg commented on HIVE-21279: [~ashutoshc] For some reason I wasn't able to create review board request. I have created a pull request at https://github.com/apache/hive/pull/552. > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.2.patch, > HIVE-21279.3.patch, HIVE-21279.4.patch, HIVE-21279.5.patch, > HIVE-21279.6.patch, HIVE-21279.7.patch, HIVE-21279.8.patch, HIVE-21279.9.patch > > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Attachment: HIVE-21292.14.patch > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21292) Break up DDLTask 1 - extract Database related operations
[ https://issues.apache.org/jira/browse/HIVE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21292: -- Status: Open (was: Patch Available) > Break up DDLTask 1 - extract Database related operations > > > Key: HIVE-21292 > URL: https://issues.apache.org/jira/browse/HIVE-21292 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21292.01.patch, HIVE-21292.02.patch, > HIVE-21292.03.patch, HIVE-21292.04.patch, HIVE-21292.05.patch, > HIVE-21292.06.patch, HIVE-21292.07.patch, HIVE-21292.08.patch, > HIVE-21292.09.patch, HIVE-21292.10.patch, HIVE-21292.11.patch, > HIVE-21292.12.patch, HIVE-21292.13.patch, HIVE-21292.14.patch > > Time Spent: 7h > Remaining Estimate: 0h > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #1: extract all the database related operations from the old DDLTask, > and move them under the new package. Also create the new internal framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Patch Available (was: Open)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21279: --- Attachment: HIVE-21279.9.patch > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.2.patch, > HIVE-21279.3.patch, HIVE-21279.4.patch, HIVE-21279.5.patch, > HIVE-21279.6.patch, HIVE-21279.7.patch, HIVE-21279.8.patch, HIVE-21279.9.patch > > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Attachment: HIVE-16924.10.patch
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Open (was: Patch Available)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch, HIVE-16924.10.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21279: --- Status: Open (was: Patch Available) > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.2.patch, > HIVE-21279.3.patch, HIVE-21279.4.patch, HIVE-21279.5.patch, > HIVE-21279.6.patch, HIVE-21279.7.patch, HIVE-21279.8.patch, HIVE-21279.9.patch > > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[
https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778607#comment-16778607
]
Hive QA commented on HIVE-21286:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16253/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16253/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive should support clean-up of previously bootstrapped tables when retry
> from different dump.
> --
>
> Key: HIVE-21286
> URL: https://issues.apache.org/jira/browse/HIVE-21286
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Attachments: HIVE-21286.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If external tables are enabled for replication on an existing repl policy,
> then bootstrapping of external tables are combined with incremental dump.
> If incremental bootstrap load fails with non-retryable error for which user
> will have to manually drop all the external tables before trying with another
> bootstrap dump. For full bootstrap, to retry with different dump, we
> suggested user to drop the DB but in this case they need to manually drop all
> the external tables which is not so user friendly. So, need to handle it in
> Hive side
[jira] [Work logged] (HIVE-20057) For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute change not reflecting for non-CAPS
[
https://issues.apache.org/jira/browse/HIVE-20057?focusedWorklogId=204794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204794
]
ASF GitHub Bot logged work on HIVE-20057:
-
Author: ASF GitHub Bot
Created on: 26/Feb/19 21:46
Start Date: 26/Feb/19 21:46
Worklog Time Spent: 10m
Work Description: rmsmani commented on issue #388: HIVE-20057: Fix Hive
table conversion DESCRIBE table bug
URL: https://github.com/apache/hive/pull/388#issuecomment-467627229
Hi @ashutosh-bapat, @sankarh
Can you merge the code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 204794)
Time Spent: 50m (was: 40m)
> For ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='TRUE'); `TBL_TYPE` attribute
> change not reflecting for non-CAPS
>
>
> Key: HIVE-20057
> URL: https://issues.apache.org/jira/browse/HIVE-20057
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: All Versions
>Reporter: Anirudh
>Assignee: Anirudh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: hive20057.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hive EXTERNAL table shown as MANAGED after conversion using
> {code} ALTER TABLE t SET TBLPROPERTIES ('EXTERNAL'='True')
> {code}
>
> The DESCRIBE FORMATTED shows:
> {code}
> Table Type: MANAGED_TABLE
> Table Parameters:
> EXTERNAL True
> {code}
>
> This is actually a External table but is shown wrongly, as 'True' was used in
> place of 'TRUE' in the ALTER statement.
> Issue explained here:
> [StakOverflow - Hive Table is MANAGED or
> EXTERNAL|https://stackoverflow.com/questions/51103317/hive-table-is-managed-or-external/51142873#51142873]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21270) A UDTF to show schema (column names and types) of given query
[
https://issues.apache.org/jira/browse/HIVE-21270?focusedWorklogId=204792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204792
]
ASF GitHub Bot logged work on HIVE-21270:
-
Author: ASF GitHub Bot
Created on: 26/Feb/19 21:45
Start Date: 26/Feb/19 21:45
Worklog Time Spent: 10m
Work Description: rmsmani commented on issue #534: HIVE-21270: A UDTF to
show schema (column names and types) of given q…
URL: https://github.com/apache/hive/pull/534#issuecomment-467626893
Hi @ashutosh-bapat, @sankarh
Can you merge the code
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 204792)
Time Spent: 1.5h (was: 1h 20m)
> A UDTF to show schema (column names and types) of given query
> -
>
> Key: HIVE-21270
> URL: https://issues.apache.org/jira/browse/HIVE-21270
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21270.1.patch, HIVE-21270.2.patch,
> HIVE-21270.3.patch, HIVE-21270.4.patch, HIVE-21270.5.patch,
> HIVE-21270.6.patch, HIVE-21270.7.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We can get ResultSet metadata using \{{ResultSet#getMetaData()}} but JDBC
> provides no way of getting nested data types(of columns) associated with it.
> This UDTF helps to retrieve each column name and it's data type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=204791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204791 ] ASF GitHub Bot logged work on HIVE-21283: - Author: ASF GitHub Bot Created on: 26/Feb/19 21:44 Start Date: 26/Feb/19 21:44 Worklog Time Spent: 10m Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms for the existing functions URL: https://github.com/apache/hive/pull/540#issuecomment-467625879 Hi @jcamachor, @ashutosh-bapat, @sankarh Can you please review the code and merge it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204791) Time Spent: 1h 10m (was: 1h) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.2.PATCH, HIVE.21283.PATCH > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=204790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204790 ] ASF GitHub Bot logged work on HIVE-21283: - Author: ASF GitHub Bot Created on: 26/Feb/19 21:43 Start Date: 26/Feb/19 21:43 Worklog Time Spent: 10m Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms for the existing functions URL: https://github.com/apache/hive/pull/540#issuecomment-467625879 Hi @jcamachor Can you please review the code and merge it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204790) Time Spent: 1h (was: 50m) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.2.PATCH, HIVE.21283.PATCH > > Time Spent: 1h > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?focusedWorklogId=204789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204789 ] ASF GitHub Bot logged work on HIVE-21283: - Author: ASF GitHub Bot Created on: 26/Feb/19 21:42 Start Date: 26/Feb/19 21:42 Worklog Time Spent: 10m Work Description: rmsmani commented on issue #540: HIVE-21283 Synonyms for the existing functions URL: https://github.com/apache/hive/pull/540#issuecomment-467625879 Hi @jcamachor, @vgarg Can you please review the code and merge it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204789) Time Spent: 50m (was: 40m) > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.2.PATCH, HIVE.21283.PATCH > > Time Spent: 50m > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21279) Avoid moving/rename operation in FileSink op for SELECT queries
[ https://issues.apache.org/jira/browse/HIVE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778597#comment-16778597 ] Ashutosh Chauhan commented on HIVE-21279: - [~vgarg] can you create RB for review? > Avoid moving/rename operation in FileSink op for SELECT queries > --- > > Key: HIVE-21279 > URL: https://issues.apache.org/jira/browse/HIVE-21279 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21279.1.patch, HIVE-21279.2.patch, > HIVE-21279.3.patch, HIVE-21279.4.patch, HIVE-21279.5.patch, > HIVE-21279.6.patch, HIVE-21279.7.patch, HIVE-21279.8.patch > > > Currently at the end of a job FileSink operator moves/rename temp directory > to another directory from which FetchTask fetches result. This is done to > avoid fetching potential partial/invalid files by failed/runway tasks. This > operation is expensive for cloud storage. It could be avoided if FetchTask is > passed on set of files to read from instead of whole directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778583#comment-16778583
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960216/HIVE-21240.11.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 14915 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=220)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestPartitionManagement - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestRawStoreProxy - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestRuntimeStats - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestTablesCreateDropAlterTruncate - did not produce a TEST-*.xml file (likely
timed out) (batchId=220)
TestTablesList - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestTxnHandlerNegative - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestTxnUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
TestUniqueConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=220)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16252/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16252/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16252/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960216 - PreCommit-HIVE-Build
> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch,
> HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch,
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch,
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch,
> kafka_storage_handler.diff
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778555#comment-16778555
]
Hive QA commented on HIVE-21240:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} serde: The patch generated 0 new + 4 unchanged - 25
fixed = 4 total (was 29) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 5
fixed = 6 total (was 11) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} serde generated 0 new + 193 unchanged - 4 fixed =
193 total (was 197) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16252/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16252/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>
[jira] [Commented] (HIVE-17879) Can not find java.sql.date in JDK9 when building hive
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778538#comment-16778538
]
Uma Maheswara Rao G commented on HIVE-17879:
Need to wait for
[https://github.com/datanucleus/datanucleus-maven-plugin/pull/11]
Once PR merged, we should update the plugin version. [~devaraj.k] verified in
his environment and confirmed its working. Thanks [~devaraj.k] for figuring out
these changes needed.
> Can not find java.sql.date in JDK9 when building hive
> -
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
>
[jira] [Reopened] (HIVE-17879) Can not find java.sql.date in JDK9 when building hive
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G reopened HIVE-17879:
> Can not find java.sql.date in JDK9 when building hive
> -
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:155)
> at org.datanucleus.plugin.PluginManager.(PluginManager.java:63)
> at
>
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778505#comment-16778505
]
Hive QA commented on HIVE-16924:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960214/HIVE-16924.09.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_1] (batchId=92)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16251/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16251/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16251/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960214 - PreCommit-HIVE-Build
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Status: Patch Available (was: Open) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.rollback.bootstrap.load= > Hive will use this config only if the current dump is bootstrap dump or > combined bootstrap in incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21286: Attachment: HIVE-21286.01.patch > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21286.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.rollback.bootstrap.load= > Hive will use this config only if the current dump is bootstrap dump or > combined bootstrap in incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778502#comment-16778502
]
Hive QA commented on HIVE-16924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 13 new + 639 unchanged - 13
fixed = 652 total (was 652) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
5s{color} | {color:red} root: The patch generated 13 new + 647 unchanged - 13
fixed = 660 total (was 660) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} ql generated 0 new + 2260 unchanged - 2 fixed = 2260
total (was 2262) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16251/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16251/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16251/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16251/yetus/whitespace-eol.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16251/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
>
[jira] [Updated] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21286: -- Labels: DR Replication pull-request-available (was: DR Replication) > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.rollback.bootstrap.load= > Hive will use this config only if the current dump is bootstrap dump or > combined bootstrap in incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21286) Hive should support clean-up of previously bootstrapped tables when retry from different dump.
[ https://issues.apache.org/jira/browse/HIVE-21286?focusedWorklogId=204708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204708 ] ASF GitHub Bot logged work on HIVE-21286: - Author: ASF GitHub Bot Created on: 26/Feb/19 19:38 Start Date: 26/Feb/19 19:38 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #551: HIVE-21286: Hive should support clean-up of previously bootstrapped tables when retry from different dump. URL: https://github.com/apache/hive/pull/551 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204708) Time Spent: 10m Remaining Estimate: 0h > Hive should support clean-up of previously bootstrapped tables when retry > from different dump. > -- > > Key: HIVE-21286 > URL: https://issues.apache.org/jira/browse/HIVE-21286 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > If external tables are enabled for replication on an existing repl policy, > then bootstrapping of external tables are combined with incremental dump. > If incremental bootstrap load fails with non-retryable error for which user > will have to manually drop all the external tables before trying with another > bootstrap dump. For full bootstrap, to retry with different dump, we > suggested user to drop the DB but in this case they need to manually drop all > the external tables which is not so user friendly. So, need to handle it in > Hive side as follows. > REPL LOAD takes additional config (passed by user in WITH clause) that says, > drop all the tables which are bootstrapped from previous dump. > hive.repl.rollback.bootstrap.load= > Hive will use this config only if the current dump is bootstrap dump or > combined bootstrap in incremental dump. > Caution to be taken by user that this config should not be passed if previous > REPL LOAD (with bootstrap) was successful or any successful incremental > dump+load happened after "previous_bootstrap_dump_dir". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778489#comment-16778489
]
BELUGA BEHR commented on HIVE-21240:
[~bslim] Can you drop the test for {{kafka_table_2}} since it is no longer
testing the 'basic implementation' as is described?
> JSON SerDe Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch,
> HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch,
> HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch,
> HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch,
> HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch,
> kafka_storage_handler.diff
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778478#comment-16778478 ] slim bouguerra commented on HIVE-21240: --- [~belugabehr] the diffs looks good to me thanks for fixing that. I still think keeping it is not an issue, it help debugging not only unit test. > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, > HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, > HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, > kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21314) Hive Replication not retaining the owner in the replicated table
[
https://issues.apache.org/jira/browse/HIVE-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778437#comment-16778437
]
Hive QA commented on HIVE-21314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960196/HIVE-21314.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15819 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16250/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16250/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16250/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960196 - PreCommit-HIVE-Build
> Hive Replication not retaining the owner in the replicated table
>
>
> Key: HIVE-21314
> URL: https://issues.apache.org/jira/browse/HIVE-21314
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21314.01.patch, HIVE-21314.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Hive Replication not retaining the owner in the replicated table. The owner
> for the target table is set same as the user executing the load command. The
> user information should be read from the dump metadata and should be used
> while creating the table at target cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21314) Hive Replication not retaining the owner in the replicated table
[
https://issues.apache.org/jira/browse/HIVE-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778245#comment-16778245
]
Hive QA commented on HIVE-21314:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 8 new + 201 unchanged - 0
fixed = 209 total (was 201) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16250/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16250/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16250/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Replication not retaining the owner in the replicated table
>
>
> Key: HIVE-21314
> URL: https://issues.apache.org/jira/browse/HIVE-21314
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21314.01.patch, HIVE-21314.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Hive Replication not retaining the owner in the replicated table. The owner
> for the target table is set same as the user executing the load command. The
> user information should be read from the dump metadata and should be used
> while creating the table at target cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778212#comment-16778212
]
Hive QA commented on HIVE-20546:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2262 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16249/dev-support/hive-personality.sh
|
| git revision | master / 20abb4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler . itests itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16249/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Apache Druid 0.13.0-incubating
> -
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch,
> HIVE-20546.3.patch, HIVE-20546.4.patch, HIVE-20546.5.patch, HIVE-20546.patch
>
>
> This task is to upgrade to druid 0.13.0 when it is released. Note that it
> will hopefully be first apache release for Druid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Apache Druid 0.13.0-incubating
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778207#comment-16778207
]
Hive QA commented on HIVE-20546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960193/HIVE-20546.5.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15778 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timeseries]
(batchId=195)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16249/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16249/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16249/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960193 - PreCommit-HIVE-Build
> Upgrade to Apache Druid 0.13.0-incubating
> -
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20546.1.patch, HIVE-20546.2.patch,
> HIVE-20546.3.patch, HIVE-20546.4.patch, HIVE-20546.5.patch, HIVE-20546.patch
>
>
> This task is to upgrade to druid 0.13.0 when it is released. Note that it
> will hopefully be first apache release for Druid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, > HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, > HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, > kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Open (was: Patch Available)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.11.patch > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, > HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, > HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, > kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.10.patch, HIVE-21240.11.patch, HIVE-21240.11.patch, > HIVE-21240.11.patch, HIVE-21240.2.patch, HIVE-21240.3.patch, > HIVE-21240.4.patch, HIVE-21240.5.patch, HIVE-21240.6.patch, > HIVE-21240.7.patch, HIVE-21240.9.patch, HIVE-24240.8.patch, > kafka_storage_handler.diff > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?focusedWorklogId=204602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204602
]
ASF GitHub Bot logged work on HIVE-16924:
-
Author: ASF GitHub Bot
Created on: 26/Feb/19 17:00
Start Date: 26/Feb/19 17:00
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #544:
HIVE-16924 Support distinct in presence of Group By
URL: https://github.com/apache/hive/pull/544#discussion_r260387274
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -4194,27 +4191,29 @@ public static long unsetBit(long bitmap, int bitIdx) {
}
/**
- * This function is a wrapper of parseInfo.getGroupByForClause which
- * automatically translates SELECT DISTINCT a,b,c to SELECT a,b,c GROUP BY
- * a,b,c.
+ * Returns the GBY, if present;
+ * DISTINCT, if present, will be handled when generating the SELECT.
*/
List getGroupByForClause(QBParseInfo parseInfo, String dest) throws
SemanticException {
-if (parseInfo.getSelForClause(dest).getToken().getType() ==
HiveParser.TOK_SELECTDI) {
- ASTNode selectExprs = parseInfo.getSelForClause(dest);
- List result = new ArrayList(selectExprs == null ? 0
- : selectExprs.getChildCount());
- if (selectExprs != null) {
-for (int i = 0; i < selectExprs.getChildCount(); ++i) {
- if (((ASTNode) selectExprs.getChild(i)).getToken().getType() ==
HiveParser.QUERY_HINT) {
+// When *not* invoked by CalcitePlanner, return the DISTINCT as a GBY
+// CBO will handle the DISTINCT in
CalcitePlannerAction.genSelectLogicalPlan
+ASTNode selectExpr = parseInfo.getSelForClause(dest);
+Collection aggregateFunction =
parseInfo.getDestToAggregationExprs().get(dest).values();
+if (isSelectDistinct(selectExpr) && !isGroupBy(selectExpr) &&
!isAggregateInSelect(selectExpr, aggregateFunction)) {
Review comment:
as we agreed, distinct with aggregate function and with group by will be
supported only if cbo is enabled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 204602)
Time Spent: 3h 10m (was: 3h)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16924) Support distinct in presence of Group By
[
https://issues.apache.org/jira/browse/HIVE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-16924:
--
Status: Patch Available (was: Open)
> Support distinct in presence of Group By
> -
>
> Key: HIVE-16924
> URL: https://issues.apache.org/jira/browse/HIVE-16924
> Project: Hive
> Issue Type: New Feature
> Components: Query Planning
>Reporter: Carter Shanklin
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-16924.01.patch, HIVE-16924.02.patch,
> HIVE-16924.03.patch, HIVE-16924.04.patch, HIVE-16924.05.patch,
> HIVE-16924.06.patch, HIVE-16924.07.patch, HIVE-16924.08.patch,
> HIVE-16924.09.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {code:sql}
> create table e011_01 (c1 int, c2 smallint);
> insert into e011_01 values (1, 1), (2, 2);
> {code}
> These queries should work:
> {code:sql}
> select distinct c1, count(*) from e011_01 group by c1;
> select distinct c1, avg(c2) from e011_01 group by c1;
> {code}
> Currently, you get :
> FAILED: SemanticException 1:52 SELECT DISTINCT and GROUP BY can not be in the
> same query. Error encountered near token 'c1'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
