[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[ https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279798#comment-16279798 ] Lefty Leverenz commented on HIVE-18127: --- Does any of this need to be documented in the wiki? > Do not strip '--' comments from shell commands issued from CliDriver > > > Key: HIVE-18127 > URL: https://issues.apache.org/jira/browse/HIVE-18127 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 3.0.0 > > Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch > > > CLiDriver has the ability to run shell commands by prefixing them with '!". > This behavior is not widely used (there are only 3 examples in .q files). > Since HIVE-16935 started stripping comments starting with '\-\-', a shell > command containing '--' will not work correctly. > Fix this by using the unstripped command for shell commands. > Note that it would be a security hole for HS2 to allow execution of arbitrary > shell commands from a client command. > Add tests to nail down correct behavior with '--' comments: > * CliDriver should not strip strings starting with '--' in a shell command > (FIXED in this change). > * HiveCli should strip '--' comments. > * A Jdbc program should allow commands starting with "!" but these will fail > in the sql parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18088) Add WM event traces at query level for debugging
[
https://issues.apache.org/jira/browse/HIVE-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279794#comment-16279794
]
Hive QA commented on HIVE-18088:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 101 new + 567 unchanged - 39
fixed = 668 total (was 606) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0d83233 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8122/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8122/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add WM event traces at query level for debugging
>
>
> Key: HIVE-18088
> URL: https://issues.apache.org/jira/browse/HIVE-18088
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18088.1.patch, HIVE-18088.2.patch,
> HIVE-18088.3.patch, HIVE-18088.4.patch, HIVE-18088.5.patch,
> HIVE-18088.6.patch, HIVE-18088.7.patch, HIVE-18088.WIP.patch
>
>
> For debugging and testing purpose, expose workload manager events via /jmx
> endpoint and print summary at the scope of query.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18222) Update checkstyle rules to be less peeky
[
https://issues.apache.org/jira/browse/HIVE-18222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279791#comment-16279791
]
Lefty Leverenz commented on HIVE-18222:
---
If the line length gets changed, the doc needs to be updated here:
* [How To Contribute -- Coding Conventions |
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CodingConventions]
> Update checkstyle rules to be less peeky
>
>
> Key: HIVE-18222
> URL: https://issues.apache.org/jira/browse/HIVE-18222
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18222.01.patch
>
>
> there are a few issues with the current checkstyle.xml
> as long as the new checks are coming back red all the time; people will start
> to ignore these checks...so I think, it would be better to make the checks
> less strict...
> * set max linelength to 140; it looks like a more natural limit - because
> there are classnames which are eating up line space pretty quickly... like:
> {{PrimitiveObjectInspector}} :)
> * make checkstyle.xml easily importable into ide (use {{config_loc}} instead
> {{basedir}})
> * suppress generated vectorized class errors
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18029) beeline - support proper usernames based on the URL arg
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279781#comment-16279781
]
Lefty Leverenz commented on HIVE-18029:
---
Okay, thanks Sergey.
> beeline - support proper usernames based on the URL arg
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18029.patch
>
>
> Update: looks like the argument connection URL is not handled consistently
> with the connect command one, and the latter passes on user name that's
> entered in the prompt correctly.
> So,
> {noformat}
> !connect (url) => prompt; the username on HS2 side is whatever is entered in
> the prompt
> beeline -u (url) => anonymous (no prompt)
> !connect (url);user=foo => foo
> beeline -u (url);user=foo => anonymous
> beeline -n foo -u (url with or without the user) => foo
> {noformat}
> I'm going to add support for extracting the user from the -u argument,
> similar to connect argument
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279777#comment-16279777
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900749/HIVE-18079.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=250)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llapdecider]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8121/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8121/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8121/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900749 - PreCommit-HIVE-Build
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch,
> HIVE-18079.4.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18206) Merge of RC/ORC file should follow other fileformate which use merge configuration parameter
[
https://issues.apache.org/jira/browse/HIVE-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279772#comment-16279772
]
Prasanth Jayachandran commented on HIVE-18206:
--
I see makes sense.
+1
> Merge of RC/ORC file should follow other fileformate which use merge
> configuration parameter
>
>
> Key: HIVE-18206
> URL: https://issues.apache.org/jira/browse/HIVE-18206
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 1.2.1, 2.1.1, 2.2.0, 3.0.0
>Reporter: Wang Haihua
>Assignee: Wang Haihua
> Attachments: HIVE-18206.1.patch, HIVE-18206.2.patch
>
>
> Merge configuration parameter, like {{hive.merge.size.per.task}} , decide the
> average file after merge stage.
> But we found it only work for file format like {{Textfile/SequenceFile}}.
> With {{RC/ORC}} file format, it {{does not work}}.
> For {{RC/ORC}} file format we found the file size after merge stage, depends
> on parameter like {{mapreduce.input.fileinputformat.split.maxsize}.
> it is better to use {{hive.merge.size.per.task}} to decide the the average
> file size for RC/ORC fileformat, which results in unifying.
> Root Cause is for RC/ORC file format, merge class is {{MergeFileTask}}
> instead of {{MapRedTask}} for Textfile/SequenceFile. And {{MergeFileTask}}
> just has not accept the configuration value in MergeFileWork, so the solution
> is passing it into {{MergeFileTask}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279770#comment-16279770 ] Prasanth Jayachandran commented on HIVE-18078: -- nit in RB. looks good otherwise, +1 pending tests. > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18078.only.patch, HIVE-18078.patch > > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18232) HDP 3.0 - Fix Hive packaging: Hive2 to be packaged as the only Hive
[ https://issues.apache.org/jira/browse/HIVE-18232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinesh Paul updated HIVE-18232: --- Target Version/s: 3.0.0 Status: Patch Available (was: Open) [~ashutoshc]. Please review the attached patch. > HDP 3.0 - Fix Hive packaging: Hive2 to be packaged as the only Hive > --- > > Key: HIVE-18232 > URL: https://issues.apache.org/jira/browse/HIVE-18232 > Project: Hive > Issue Type: Bug >Reporter: Vinesh Paul > Attachments: HIVE-18232.1.patch > > > As discussed with Ashutosh Chauhan this change is to include init-hive-dfs.sh > in the hive rpm package. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18232) HDP 3.0 - Fix Hive packaging: Hive2 to be packaged as the only Hive
[ https://issues.apache.org/jira/browse/HIVE-18232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinesh Paul updated HIVE-18232: --- Attachment: HIVE-18232.1.patch > HDP 3.0 - Fix Hive packaging: Hive2 to be packaged as the only Hive > --- > > Key: HIVE-18232 > URL: https://issues.apache.org/jira/browse/HIVE-18232 > Project: Hive > Issue Type: Bug >Reporter: Vinesh Paul > Attachments: HIVE-18232.1.patch > > > As discussed with Ashutosh Chauhan this change is to include init-hive-dfs.sh > in the hive rpm package. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18211) Support to read multiple level definition for Map type in Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279741#comment-16279741
]
Colin Ma commented on HIVE-18211:
-
[~Ferd], can you help to review the patch, thanks.
> Support to read multiple level definition for Map type in Parquet file
> --
>
> Key: HIVE-18211
> URL: https://issues.apache.org/jira/browse/HIVE-18211
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18211.001.patch
>
>
> For the current implementation with VectorizedParquetRecordReader, only
> following definition for map type is supported:
> {code}
> repeated group map (MAP_KEY_VALUE) {
> required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
> The implementation should support multiple level definition like:
> {code}
> optional group m1 (MAP) {
> repeated group map (MAP_KEY_VALUE)
> {required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18211) Support to read multiple level definition for Map type in Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated HIVE-18211:
Status: Patch Available (was: Open)
> Support to read multiple level definition for Map type in Parquet file
> --
>
> Key: HIVE-18211
> URL: https://issues.apache.org/jira/browse/HIVE-18211
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18211.001.patch
>
>
> For the current implementation with VectorizedParquetRecordReader, only
> following definition for map type is supported:
> {code}
> repeated group map (MAP_KEY_VALUE) {
> required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
> The implementation should support multiple level definition like:
> {code}
> optional group m1 (MAP) {
> repeated group map (MAP_KEY_VALUE)
> {required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18211) Support to read multiple level definition for Map type in Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated HIVE-18211:
Attachment: HIVE-18211.001.patch
> Support to read multiple level definition for Map type in Parquet file
> --
>
> Key: HIVE-18211
> URL: https://issues.apache.org/jira/browse/HIVE-18211
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18211.001.patch
>
>
> For the current implementation with VectorizedParquetRecordReader, only
> following definition for map type is supported:
> {code}
> repeated group map (MAP_KEY_VALUE) {
> required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
> The implementation should support multiple level definition like:
> {code}
> optional group m1 (MAP) {
> repeated group map (MAP_KEY_VALUE)
> {required binary key (UTF8); optional binary value (UTF8);}
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18173) Improve plans for correlated subqueries with non-equi predicate
[
https://issues.apache.org/jira/browse/HIVE-18173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18173:
---
Status: Open (was: Patch Available)
> Improve plans for correlated subqueries with non-equi predicate
> ---
>
> Key: HIVE-18173
> URL: https://issues.apache.org/jira/browse/HIVE-18173
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18173.1.patch, HIVE-18173.2.patch,
> HIVE-18173.3.patch, HIVE-18173.4.patch, HIVE-18173.5.patch, HIVE-18173.6.patch
>
>
> HIVE-17767 optimized plan to not generate value generator (i.e. an extra join
> with outer query to fetch correlated columns) for EQUAL and NOT EQUAL
> predicates e.g.
> {code:sql}
> select * from src b where b.key in (select key from src a where b.value <>
> a.value)
> {code}
> This should be improved and implemented for rest of the predicates e.g. LESS
> THAN etc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18173) Improve plans for correlated subqueries with non-equi predicate
[
https://issues.apache.org/jira/browse/HIVE-18173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18173:
---
Status: Patch Available (was: Open)
> Improve plans for correlated subqueries with non-equi predicate
> ---
>
> Key: HIVE-18173
> URL: https://issues.apache.org/jira/browse/HIVE-18173
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18173.1.patch, HIVE-18173.2.patch,
> HIVE-18173.3.patch, HIVE-18173.4.patch, HIVE-18173.5.patch, HIVE-18173.6.patch
>
>
> HIVE-17767 optimized plan to not generate value generator (i.e. an extra join
> with outer query to fetch correlated columns) for EQUAL and NOT EQUAL
> predicates e.g.
> {code:sql}
> select * from src b where b.key in (select key from src a where b.value <>
> a.value)
> {code}
> This should be improved and implemented for rest of the predicates e.g. LESS
> THAN etc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18173) Improve plans for correlated subqueries with non-equi predicate
[
https://issues.apache.org/jira/browse/HIVE-18173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18173:
---
Attachment: HIVE-18173.6.patch
> Improve plans for correlated subqueries with non-equi predicate
> ---
>
> Key: HIVE-18173
> URL: https://issues.apache.org/jira/browse/HIVE-18173
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-18173.1.patch, HIVE-18173.2.patch,
> HIVE-18173.3.patch, HIVE-18173.4.patch, HIVE-18173.5.patch, HIVE-18173.6.patch
>
>
> HIVE-17767 optimized plan to not generate value generator (i.e. an extra join
> with outer query to fetch correlated columns) for EQUAL and NOT EQUAL
> predicates e.g.
> {code:sql}
> select * from src b where b.key in (select key from src a where b.value <>
> a.value)
> {code}
> This should be improved and implemented for rest of the predicates e.g. LESS
> THAN etc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279707#comment-16279707
]
Hive QA commented on HIVE-18079:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} standalone-metastore: The patch generated 1 new + 41
unchanged - 0 fixed = 42 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0d83233 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8121/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8121/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8121/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch,
> HIVE-18079.4.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18209) Fix API call in VectorizedListColumnReader to get value from BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279692#comment-16279692 ] Colin Ma commented on HIVE-18209: - Update the patch and add test case to verify the ListColumnVector.isRepeating. > Fix API call in VectorizedListColumnReader to get value from BytesColumnVector > -- > > Key: HIVE-18209 > URL: https://issues.apache.org/jira/browse/HIVE-18209 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18209.001.patch, HIVE-18209.002.patch, > HIVE-18209.003.patch > > > With the API BytesColumnVector.setVal(), the isRepeating attribute can't be > set correctly if ListColumnVector.child is BytesColumnVector. > BytesColumnVector.setRef() should be used to avoid this problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18209) Fix API call in VectorizedListColumnReader to get value from BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18209: Description: With the API BytesColumnVector.setVal(), the isRepeating attribute can't be set correctly if ListColumnVector.child is BytesColumnVector. BytesColumnVector.setRef() should be used to avoid this problem. (was: BytesColumnVector.setRef() should be instead of BytesColumnVector.setVal() to get the result as expected.) > Fix API call in VectorizedListColumnReader to get value from BytesColumnVector > -- > > Key: HIVE-18209 > URL: https://issues.apache.org/jira/browse/HIVE-18209 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18209.001.patch, HIVE-18209.002.patch, > HIVE-18209.003.patch > > > With the API BytesColumnVector.setVal(), the isRepeating attribute can't be > set correctly if ListColumnVector.child is BytesColumnVector. > BytesColumnVector.setRef() should be used to avoid this problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18209) Fix API call in VectorizedListColumnReader to get value from BytesColumnVector
[ https://issues.apache.org/jira/browse/HIVE-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18209: Attachment: HIVE-18209.003.patch > Fix API call in VectorizedListColumnReader to get value from BytesColumnVector > -- > > Key: HIVE-18209 > URL: https://issues.apache.org/jira/browse/HIVE-18209 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18209.001.patch, HIVE-18209.002.patch, > HIVE-18209.003.patch > > > BytesColumnVector.setRef() should be instead of BytesColumnVector.setVal() to > get the result as expected. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18153) refactor reopen and file management in TezTask
[
https://issues.apache.org/jira/browse/HIVE-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279687#comment-16279687
]
Hive QA commented on HIVE-18153:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900739/HIVE-18153.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 148 failed/errored test(s), 10830 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
(batchId=103)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query10]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query11]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query12]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query13]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query15]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query17]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query18]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query19]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query1]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query20]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query21]
(batchId=247)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query22]
(batchId=247)
[jira] [Commented] (HIVE-18153) refactor reopen and file management in TezTask
[
https://issues.apache.org/jira/browse/HIVE-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279660#comment-16279660
]
Hive QA commented on HIVE-18153:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 40 new + 429 unchanged - 38
fixed = 469 total (was 467) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0d83233 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8120/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8120/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> refactor reopen and file management in TezTask
> --
>
> Key: HIVE-18153
> URL: https://issues.apache.org/jira/browse/HIVE-18153
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18153.01.patch, HIVE-18153.patch
>
>
> TezTask reopen relies on getting the same session object in terms of setup;
> WM reopen returns a new session from the pool.
> The former has the advantage of not having to reupload files and stuff... but
> the object reuse results in a lot of ugly code, and also reopen might be
> slower on average with the session pool than just getting a session from the
> pool. Either WM needs to do the object-preserving reopen, or TezTask needs to
> be refactored. It looks like DAG would have to be rebuilt to do the latter
> because of some paths tied to a directory of the old session. Let me see if I
> can get around that; if not we can do the former; and then if the former
> results in too much ugly code in WM to account for object reuse for different
> Tez client I'd do the latter anyway since it's a failure path :)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279642#comment-16279642
]
Hive QA commented on HIVE-15939:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900702/HIVE-15939.07.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8119/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8119/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8119/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900702 - PreCommit-HIVE-Build
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18210) create resource plan allows duplicates
[
https://issues.apache.org/jira/browse/HIVE-18210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279633#comment-16279633
]
Sergey Shelukhin commented on HIVE-18210:
-
+1
> create resource plan allows duplicates
> --
>
> Key: HIVE-18210
> URL: https://issues.apache.org/jira/browse/HIVE-18210
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18210.01.patch
>
>
> Create resource plan allows duplicates. This was seen in a cluster:
> {code}
> +--+---++
> | rp_name | status | query_parallelism |
> +--+---++
> | plan_2 | ACTIVE| 10 |
> | plan_2 | DISABLED | NULL |
> +--+---++
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18210) create resource plan allows duplicates
[
https://issues.apache.org/jira/browse/HIVE-18210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279623#comment-16279623
]
Harish Jaiprakash commented on HIVE-18210:
--
[~sershe], This should fix duplicate resource plan names. I'll look into rest
of the issues.
> create resource plan allows duplicates
> --
>
> Key: HIVE-18210
> URL: https://issues.apache.org/jira/browse/HIVE-18210
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18210.01.patch
>
>
> Create resource plan allows duplicates. This was seen in a cluster:
> {code}
> +--+---++
> | rp_name | status | query_parallelism |
> +--+---++
> | plan_2 | ACTIVE| 10 |
> | plan_2 | DISABLED | NULL |
> +--+---++
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18004) investigate deriving app name from JDBC connection for pool mapping
[ https://issues.apache.org/jira/browse/HIVE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18004: --- Assignee: Sergey Shelukhin > investigate deriving app name from JDBC connection for pool mapping > --- > > Key: HIVE-18004 > URL: https://issues.apache.org/jira/browse/HIVE-18004 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > There are some client info fields that popular apps (Tableau, etc) might > populate; this might allow us to map queries to pools based on an application > used. Need to take a look (see the doc for an example API we might look into) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18003) add explicit jdbc connection string args for mappings
[ https://issues.apache.org/jira/browse/HIVE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18003: Attachment: HIVE-18003.patch > add explicit jdbc connection string args for mappings > - > > Key: HIVE-18003 > URL: https://issues.apache.org/jira/browse/HIVE-18003 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18003.patch > > > 1) Force using unmanaged/containers execution. > 2) Optional - specify pool name (config setting to gate this, disabled by > default?). > In phase 2 (or 4?) we might allow #2 to be used by a user to choose between > multiple mappings if they have multiple pools they could be mapped to (i.e. > to change the ordering essentially). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18003) add explicit jdbc connection string args for mappings
[ https://issues.apache.org/jira/browse/HIVE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18003: Attachment: (was: HIVE-18003.patch) > add explicit jdbc connection string args for mappings > - > > Key: HIVE-18003 > URL: https://issues.apache.org/jira/browse/HIVE-18003 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18003.patch > > > 1) Force using unmanaged/containers execution. > 2) Optional - specify pool name (config setting to gate this, disabled by > default?). > In phase 2 (or 4?) we might allow #2 to be used by a user to choose between > multiple mappings if they have multiple pools they could be mapped to (i.e. > to change the ordering essentially). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18003) add explicit jdbc connection string args for mappings
[ https://issues.apache.org/jira/browse/HIVE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18003: Attachment: HIVE-18003.patch > add explicit jdbc connection string args for mappings > - > > Key: HIVE-18003 > URL: https://issues.apache.org/jira/browse/HIVE-18003 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18003.patch > > > 1) Force using unmanaged/containers execution. > 2) Optional - specify pool name (config setting to gate this, disabled by > default?). > In phase 2 (or 4?) we might allow #2 to be used by a user to choose between > multiple mappings if they have multiple pools they could be mapped to (i.e. > to change the ordering essentially). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18003) add explicit jdbc connection string args for mappings
[ https://issues.apache.org/jira/browse/HIVE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18003: Status: Patch Available (was: Open) [~prasanth_j] can you take a look? > add explicit jdbc connection string args for mappings > - > > Key: HIVE-18003 > URL: https://issues.apache.org/jira/browse/HIVE-18003 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18003.patch > > > 1) Force using unmanaged/containers execution. > 2) Optional - specify pool name (config setting to gate this, disabled by > default?). > In phase 2 (or 4?) we might allow #2 to be used by a user to choose between > multiple mappings if they have multiple pools they could be mapped to (i.e. > to change the ordering essentially). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279580#comment-16279580
]
Hive QA commented on HIVE-15939:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} serde: The patch generated 0 new + 192 unchanged - 7
fixed = 192 total (was 199) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} ql: The patch generated 0 new + 444 unchanged - 48
fixed = 444 total (was 492) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0d83233 |
| Default Java | 1.8.0_111 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8119/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18185) update insert_values_orig_table_use_metadata.q.out
[
https://issues.apache.org/jira/browse/HIVE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279558#comment-16279558
]
Hive QA commented on HIVE-18185:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900735/HIVE-18185.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11509 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testPartitionOps
(batchId=202)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8118/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8118/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8118/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900735 - PreCommit-HIVE-Build
> update insert_values_orig_table_use_metadata.q.out
> --
>
> Key: HIVE-18185
> URL: https://issues.apache.org/jira/browse/HIVE-18185
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Bertalan Kondrat
> Attachments: HIVE-18185.patch
>
>
> some time back this test was skipped; and put back in: HIVE-17076
> however...possibly some other changes were also present around that time;
> which have caused a 1 byte difference in totalSize; and after that some stats
> changes came by (HIVE-16827 / others)?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-18052: -- Attachment: (was: HIVE-18052.6.patch) > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Steve Yeom > Attachments: HIVE-18052.1.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18150) Upgrade Spark Version to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-18150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang updated HIVE-18150: -- Attachment: (was: HIVE-18150.1.patch) > Upgrade Spark Version to 2.2.0 > -- > > Key: HIVE-18150 > URL: https://issues.apache.org/jira/browse/HIVE-18150 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: liyunzhang > Attachments: HIVE-18150.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18150) Upgrade Spark Version to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-18150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang updated HIVE-18150: -- Attachment: HIVE-18150.1.patch reattach patch to trigger Hive QA. > Upgrade Spark Version to 2.2.0 > -- > > Key: HIVE-18150 > URL: https://issues.apache.org/jira/browse/HIVE-18150 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Sahil Takiar >Assignee: liyunzhang > Attachments: HIVE-18150.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-18052: -- Attachment: HIVE-18052.6.patch > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Steve Yeom > Attachments: HIVE-18052.1.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, HIVE-18052.6.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279486#comment-16279486 ] Ashutosh Chauhan commented on HIVE-18149: - +1 > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18149.01.patch, HIVE-18149.01wip01.patch > > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18189) Order by position does not work when cbo is disabled
[
https://issues.apache.org/jira/browse/HIVE-18189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279471#comment-16279471
]
Ashutosh Chauhan commented on HIVE-18189:
-
+1
> Order by position does not work when cbo is disabled
>
>
> Key: HIVE-18189
> URL: https://issues.apache.org/jira/browse/HIVE-18189
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18189.1.patch, HIVE-18189.2.patch,
> HIVE-18189.3.patch
>
>
> Investigating a failed query:
> {code}
> set hive.cbo.enable=false;
> set hive.orderby.position.alias=true;
> select distinct age from student order by 1 desc limit 20;
> {code}
> The query does not sort the output correctly when cbo is
> disabled/inactivated. I found two issues:
> 1. "order by position" query is broken by HIVE-16774
> 2. In particular, select distinct query never work for "order by position"
> query
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18185) update insert_values_orig_table_use_metadata.q.out
[
https://issues.apache.org/jira/browse/HIVE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279465#comment-16279465
]
Hive QA commented on HIVE-18185:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0d83233 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8118/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> update insert_values_orig_table_use_metadata.q.out
> --
>
> Key: HIVE-18185
> URL: https://issues.apache.org/jira/browse/HIVE-18185
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Bertalan Kondrat
> Attachments: HIVE-18185.patch
>
>
> some time back this test was skipped; and put back in: HIVE-17076
> however...possibly some other changes were also present around that time;
> which have caused a 1 byte difference in totalSize; and after that some stats
> changes came by (HIVE-16827 / others)?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[
https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279445#comment-16279445
]
Hive QA commented on HIVE-18149:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900726/HIVE-18149.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11509 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_character_length]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_octet_length]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_view] (batchId=15)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_view]
(batchId=110)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8117/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8117/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900726 - PreCommit-HIVE-Build
> Stats: rownum estimation from datasize underestimates in most cases
> ---
>
> Key: HIVE-18149
> URL: https://issues.apache.org/jira/browse/HIVE-18149
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18149.01.patch, HIVE-18149.01wip01.patch
>
>
> rownum estimation is based on the following fact as of now:
> * datasize being used from the following sources:
> ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are
> able to give "raw size" estimation - I've checked orc; but I'm sure others
> will do the sameapi docs are a bit vague about the methods purpose...
> ** if the basicstats level info is not available; the filesystem level
> "file-size-sums" are used as the "raw data size" ; which is multiplied by the
> [deserialization
> ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261]
> ; which is currently 1.
> the problem with all of this is that deser factor is 1; and that rowsize
> counts in the online object headers..
> example; 20 rows are loaded into a partition
> [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7]
> after HIVE-18108 [this
> explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25]
> will estimate the rowsize of the table to be 404 bytes; however the 20 rows
> of text is only 169 bytes...so it ends up with 0 rows...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (HIVE-18227) Tez parallel execution fail
[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18227:
---
Comment: was deleted
(was: Looking at the patch, this is converting {{hive.exec.parallel=true}} into
a single threaded operation by use of a mutex section - this flag is false by
default.)
> Tez parallel execution fail
> ---
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18227.1.patch
>
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=10;
> set tez.grouping.min-size=1;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18227) Tez parallel execution fail
[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279431#comment-16279431
]
Gopal V commented on HIVE-18227:
Looking at the patch, this is converting {{hive.exec.parallel=true}} into a
single threaded operation by use of a mutex section - this flag is false by
default.
> Tez parallel execution fail
> ---
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18227.1.patch
>
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=10;
> set tez.grouping.min-size=1;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18003) add explicit jdbc connection string args for mappings
[ https://issues.apache.org/jira/browse/HIVE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18003: --- Assignee: Sergey Shelukhin > add explicit jdbc connection string args for mappings > - > > Key: HIVE-18003 > URL: https://issues.apache.org/jira/browse/HIVE-18003 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > 1) Force using unmanaged/containers execution. > 2) Optional - specify pool name (config setting to gate this, disabled by > default?). > In phase 2 (or 4?) we might allow #2 to be used by a user to choose between > multiple mappings if they have multiple pools they could be mapped to (i.e. > to change the ordering essentially). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18138) Fix columnstats problem in case schema evolution
[ https://issues.apache.org/jira/browse/HIVE-18138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279421#comment-16279421 ] Ashutosh Chauhan commented on HIVE-18138: - +1 > Fix columnstats problem in case schema evolution > > > Key: HIVE-18138 > URL: https://issues.apache.org/jira/browse/HIVE-18138 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18138.01.patch, HIVE-18138.01.patch, > HIVE-18138.02.patch > > > column stats are kept in case the main table schema is altered; and this > causes all kind of problems. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279417#comment-16279417 ] Sergey Shelukhin commented on HIVE-18231: - cc [~prasanth_j] [~harishjp] > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-16821: --- Status: Patch Available (was: Open) Rebased to master + modifies the vectorForward() impl to support PTF operator > Vectorization: support Explain Analyze in vectorized mode > - > > Key: HIVE-16821 > URL: https://issues.apache.org/jira/browse/HIVE-16821 > Project: Hive > Issue Type: Bug > Components: Diagnosability, Vectorization >Affects Versions: 2.1.1, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch, > HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch > > > Currently, to avoid a branch in the operator inner loop - the runtime stats > are only available in non-vector mode. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16821) Vectorization: support Explain Analyze in vectorized mode
[ https://issues.apache.org/jira/browse/HIVE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-16821: --- Attachment: HIVE-16821.7.patch > Vectorization: support Explain Analyze in vectorized mode > - > > Key: HIVE-16821 > URL: https://issues.apache.org/jira/browse/HIVE-16821 > Project: Hive > Issue Type: Bug > Components: Diagnosability, Vectorization >Affects Versions: 2.1.1, 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Minor > Attachments: HIVE-16821.1.patch, HIVE-16821.2.patch, > HIVE-16821.2.patch, HIVE-16821.3.patch, HIVE-16821.7.patch > > > Currently, to avoid a branch in the operator inner loop - the runtime stats > are only available in non-vector mode. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18179) Implement validate resource plan (part 1)
[ https://issues.apache.org/jira/browse/HIVE-18179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18179: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Will file a follow-up to validate the expressions > Implement validate resource plan (part 1) > - > > Key: HIVE-18179 > URL: https://issues.apache.org/jira/browse/HIVE-18179 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Fix For: 3.0.0 > > Attachments: HIVE-18179.01.patch, HIVE-18179.02.patch, > HIVE-18179.03.patch > > > For now mostly pool alloc fraction to add up to 1.0 and some null or invalid > value checks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279410#comment-16279410
]
Eugene Koifman commented on HIVE-18221:
---
lots of TestCLIDriver failures due to
{noformat}
FAILED: SemanticException [Error 30023]: alltypesorc file name is not valid in
Load Data into Acid table default.alltypesorc. Examples of valid names are:
0_0, 0_0_copy_1
{noformat}
lots of HCat failures
lots of
{noformat}
java.lang.RuntimeException: Unexpected exception in static initialization
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:90)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:71)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1305)
at
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1281)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1469)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1594)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1358)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1346)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:342)
at org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:999)
at org.apache.hadoop.hive.ql.QTestUtil.cleanUp(QTestUtil.java:975)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver$2.invokeInternal(CoreCliDriver.java:73)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver$2.invokeInternal(CoreCliDriver.java:70)
at
org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.beforeClass(CoreCliDriver.java:76)
... 11 more
{noformat}
bunch of failures due to
{noformat}
017-12-05T14:23:01,653 INFO [main] ql.Driver: Compiling
command(queryId=hiveptest_20171205142257_78d50ca8-c29c-422f-8553-f6f9b5c0250b):
DROP TABLE IF EXISTS cmd_testdb.cmd_testtable FOR REPLICATION('157')
FAILED: RuntimeException [Error 10264]: To use DbTxnManager you must set
hive.support.concurrency=true
2017-12-05T14:23:01,654 ERROR [main] ql.Driver: FAILED: RuntimeException [Error
10264]: To use DbTxnManager you must set hive.support.concurrency=true
java.lang.RuntimeException: To use DbTxnManager you must set
hive.support.concurrency=true
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.setHiveConf(DbTxnManager.java:189)
at
org.apache.hadoop.hive.ql.lockmgr.TxnManagerFactory.getTxnManager(TxnManagerFactory.java:72)
at
org.apache.hadoop.hive.ql.session.SessionState.initTxnMgr(SessionState.java:463)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:519)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1466)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1594)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1358)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1346)
at
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testDropTableCommand2(TestCommands.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
{noformat}
which then lead to
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1305)
at
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1281)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1469)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1594)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1358)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1346)
at
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testDropTableCommand2(TestCommands.java:355)
{noformat}
patch 4 should get past
2017-12-05T14:23:01,654 ERROR [main] ql.Driver: FAILED: RuntimeException [Error
10264]: To use DbTxnManager you must set hive.support.concurrency=true
hvie.txn.manager and hive.support.concurrency are currently in both HiveConf
and MetastoreConf - the later was not modified in patch 3
> test acid default
> -
>
> Key: HIVE-18221
> URL:
[jira] [Assigned] (HIVE-18096) add a user-friendly show plan command
[ https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18096: --- Assignee: Harish Jaiprakash > add a user-friendly show plan command > - > > Key: HIVE-18096 > URL: https://issues.apache.org/jira/browse/HIVE-18096 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > > For admin to be able to get an overview of a resource plan. > We need to try to do this using sysdb. > If that is not possible to do in a nice way, we'd do a text-based one like > query explain, or desc extended table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279397#comment-16279397 ] Sergey Shelukhin commented on HIVE-18075: - At least 2 and 3 still apply. > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # -Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason-). > # -Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands-. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18075: --- Assignee: Harish Jaiprakash > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # -Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason-). > # -Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands-. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18179) Implement validate resource plan (part 1)
[ https://issues.apache.org/jira/browse/HIVE-18179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18179: Summary: Implement validate resource plan (part 1) (was: Implement validate resource plan.) > Implement validate resource plan (part 1) > - > > Key: HIVE-18179 > URL: https://issues.apache.org/jira/browse/HIVE-18179 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-18179.01.patch, HIVE-18179.02.patch, > HIVE-18179.03.patch > > > For now mostly pool alloc fraction to add up to 1.0 and some null or invalid > value checks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Attachment: HIVE-18221.04.patch > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18078: Attachment: HIVE-18078.patch The combined patch for HiveQA > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18078.only.patch, HIVE-18078.patch > > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18078: Attachment: HIVE-18078.only.patch A small patch, on top of the reopen patch but I will rebase it on master if that one takes too long. > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18078.only.patch, HIVE-18078.patch > > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18078: Status: Patch Available (was: Open) > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18078.only.patch, HIVE-18078.patch > > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[
https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279361#comment-16279361
]
Hive QA commented on HIVE-18149:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fb85336 |
| Default Java | 1.8.0_111 |
| modules | C: common ql contrib itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8117/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Stats: rownum estimation from datasize underestimates in most cases
> ---
>
> Key: HIVE-18149
> URL: https://issues.apache.org/jira/browse/HIVE-18149
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18149.01.patch, HIVE-18149.01wip01.patch
>
>
> rownum estimation is based on the following fact as of now:
> * datasize being used from the following sources:
> ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are
> able to give "raw size" estimation - I've checked orc; but I'm sure others
> will do the sameapi docs are a bit vague about the methods purpose...
> ** if the basicstats level info is not available; the filesystem level
> "file-size-sums" are used as the "raw data size" ; which is multiplied by the
> [deserialization
> ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261]
> ; which is currently 1.
> the problem with all of this is that deser factor is 1; and that rowsize
> counts in the online object headers..
> example; 20 rows are loaded into a partition
> [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7]
> after HIVE-18108 [this
>
[jira] [Commented] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279356#comment-16279356
]
Ashutosh Chauhan commented on HIVE-15939:
-
+1 pending tests
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18228) Azure credential properties should be added to the HiveConf hidden list
[
https://issues.apache.org/jira/browse/HIVE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18228:
-
> Azure credential properties should be added to the HiveConf hidden list
> ---
>
> Key: HIVE-18228
> URL: https://issues.apache.org/jira/browse/HIVE-18228
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>
> The HIVE_CONF_HIDDEN_LIST("hive.conf.hidden.list") already contains keys
> contaiing aws credentials. The Azure properties to be added are:
> * dfs.adls.oauth2.credential
> * fs.adl.oauth2.credential
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279331#comment-16279331 ] Ashutosh Chauhan commented on HIVE-18149: - Since ORC and parquet are most common formats these days, bumping up this ratio makes sense, since columnar formats usually compresses very well and then there is bloat in memory size after this as well. > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18149.01.patch, HIVE-18149.01wip01.patch > > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18166) Result of hive.query.string is encoded.
[ https://issues.apache.org/jira/browse/HIVE-18166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279327#comment-16279327 ] Sahil Takiar commented on HIVE-18166: - LGTM +1 > Result of hive.query.string is encoded. > --- > > Key: HIVE-18166 > URL: https://issues.apache.org/jira/browse/HIVE-18166 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-18166.1.patch, HIVE-18166.2.patch > > > set hive.query.string returns encoded string. > hive.query.string=%0A%0Aselect+*+from+t1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17980) Move HiveMetaStoreClient plus a few remaining classes.
[ https://issues.apache.org/jira/browse/HIVE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279324#comment-16279324 ] Daniel Dai commented on HIVE-17980: --- Need some rebase, otherwise +1. > Move HiveMetaStoreClient plus a few remaining classes. > -- > > Key: HIVE-17980 > URL: https://issues.apache.org/jira/browse/HIVE-17980 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Attachments: HIVE-17980.2.patch, HIVE-17980.patch > > > This moves over HiveMetaStoreClient a few more remaining things that haven't > been moved yet. It also cleans up the mess I made of MetaStoreUtils. I > renamed the class in metastore to HiveMetaStoreUtils and left a few methods > in there that could be moved. Everything else has been moved to > standalone-metastore/.../MetaStoreUtils. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279322#comment-16279322
]
Hive QA commented on HIVE-18221:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900722/HIVE-18221.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 891 failed/errored test(s), 6662 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key2]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_custom_key]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=239)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=239)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=247)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=250)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=250)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_orcfile]
(batchId=250)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
[jira] [Updated] (HIVE-18227) Tez parallel execution fail
[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-18227:
--
Attachment: HIVE-18227.1.patch
> Tez parallel execution fail
> ---
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18227.1.patch
>
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=10;
> set tez.grouping.min-size=1;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18227) Tez parallel execution fail
[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-18227:
--
Status: Patch Available (was: Open)
> Tez parallel execution fail
> ---
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18227.1.patch
>
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=10;
> set tez.grouping.min-size=1;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18227) Tez parallel execution fail
[
https://issues.apache.org/jira/browse/HIVE-18227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-18227:
-
> Tez parallel execution fail
> ---
>
> Key: HIVE-18227
> URL: https://issues.apache.org/jira/browse/HIVE-18227
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> Running tez Dag in parallel within a session fail. Here is the test case:
> {code}
> set hive.exec.parallel=true;
> set hive.merge.tezfiles=true;
> set tez.grouping.max-size=10;
> set tez.grouping.min-size=1;
> from student
> insert overwrite table student4 select *
> insert overwrite table student5 select *
> insert overwrite table student6 select *;
> {code}
> The merge task run in parallel and result the exception:
> {code}
> org.apache.tez.dag.api.TezException: App master already running a DAG
> at
> org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1255)
> at
> org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:161)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17716) Not pushing postaggregations into Druid due to CAST on constant
[
https://issues.apache.org/jira/browse/HIVE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279285#comment-16279285
]
slim bouguerra commented on HIVE-17716:
---
Here another example
{code}
PREHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(cdouble) + 1.0
FROM druid_table GROUP BY floor_year(`__time`)
PREHOOK: type: QUERY
POSTHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(cdouble) + 1.0
FROM druid_table GROUP BY floor_year(`__time`)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: druid_table
properties:
druid.query.json
{"queryType":"timeseries","dataSource":"default.druid_table","descending":false,"granularity":"year","aggregations":[{"type":"doubleSum","name":"$f1","fieldName":"cdouble"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"],"context":{"skipEmptyBuckets":true}}
druid.query.type timeseries
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL Column
stats: NONE
Select Operator
expressions: __time (type: timestamp with local time zone), ($f1
+ 1.0) (type: double)
outputColumnNames: _col0, _col1
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
Column stats: NONE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
{code}
> Not pushing postaggregations into Druid due to CAST on constant
> ---
>
> Key: HIVE-17716
> URL: https://issues.apache.org/jira/browse/HIVE-17716
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Jesus Camacho Rodriguez
>
> After Calcite is upgraded to 1.14 and the rule to push post-aggregations to
> Druid is enabled, the following query fails to create a postaggregation:
> {code}
> EXPLAIN
> SELECT language, sum(added) + 100 AS a
> FROM druid_table_1
> GROUP BY language
> ORDER BY a DESC;
> {code}
> Problem seems to be that CAST is getting on the way for the rule to be
> applied. In particular, this is the final Calcite plan:
> {code}
> HiveSortLimit(sort0=[$1], dir0=[DESC-nulls-last])
> HiveProject(language=[$0], a=[+($1, CAST(100):DOUBLE)])
> DruidQuery(table=[[default.druid_table_1]],
> intervals=[[1900-01-01T00:00:00.000/3000-01-01T00:00:00.000]], groups=[{6}],
> aggs=[[sum($10)]])
> {code}
> There are two different parts to explore to seek a solution: 1) why
> {{CAST(100):DOUBLE)}} is not folded to {{100.0d}}, and 2) whether the rule to
> push post-aggregations to Druid could handle the CAST in some particular
> cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17716) Not pushing postaggregations into Druid due to CAST on constant
[
https://issues.apache.org/jira/browse/HIVE-17716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279284#comment-16279284
]
slim bouguerra commented on HIVE-17716:
---
Even without cast I still see that the project is not pushed down as per the
following example.
{code}
PREHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(cint) + 1
FROM druid_table GROUP BY floor_year(`__time`)
PREHOOK: type: QUERY
POSTHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(cint) + 1
FROM druid_table GROUP BY floor_year(`__time`)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: druid_table
properties:
druid.query.json
{"queryType":"timeseries","dataSource":"default.druid_table","descending":false,"granularity":"year","aggregations":[{"type":"longSum","name":"$f1","f
ieldName":"cint"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"],"context":{"skipEmptyBuckets":true}}
druid.query.type timeseries
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL Column
stats: NONE
Select Operator
expressions: __time (type: timestamp with local time zone), ($f1
+ 1) (type: bigint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
Column stats: NONE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
{code}
> Not pushing postaggregations into Druid due to CAST on constant
> ---
>
> Key: HIVE-17716
> URL: https://issues.apache.org/jira/browse/HIVE-17716
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Jesus Camacho Rodriguez
>
> After Calcite is upgraded to 1.14 and the rule to push post-aggregations to
> Druid is enabled, the following query fails to create a postaggregation:
> {code}
> EXPLAIN
> SELECT language, sum(added) + 100 AS a
> FROM druid_table_1
> GROUP BY language
> ORDER BY a DESC;
> {code}
> Problem seems to be that CAST is getting on the way for the rule to be
> applied. In particular, this is the final Calcite plan:
> {code}
> HiveSortLimit(sort0=[$1], dir0=[DESC-nulls-last])
> HiveProject(language=[$0], a=[+($1, CAST(100):DOUBLE)])
> DruidQuery(table=[[default.druid_table_1]],
> intervals=[[1900-01-01T00:00:00.000/3000-01-01T00:00:00.000]], groups=[{6}],
> aggs=[[sum($10)]])
> {code}
> There are two different parts to explore to seek a solution: 1) why
> {{CAST(100):DOUBLE)}} is not folded to {{100.0d}}, and 2) whether the rule to
> push post-aggregations to Druid could handle the CAST in some particular
> cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18088) Add WM event traces at query level for debugging
[ https://issues.apache.org/jira/browse/HIVE-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18088: - Attachment: HIVE-18088.7.patch Last patch was not picked by precommit , so addressed the nit. > Add WM event traces at query level for debugging > > > Key: HIVE-18088 > URL: https://issues.apache.org/jira/browse/HIVE-18088 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18088.1.patch, HIVE-18088.2.patch, > HIVE-18088.3.patch, HIVE-18088.4.patch, HIVE-18088.5.patch, > HIVE-18088.6.patch, HIVE-18088.7.patch, HIVE-18088.WIP.patch > > > For debugging and testing purpose, expose workload manager events via /jmx > endpoint and print summary at the scope of query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279263#comment-16279263
]
Hive QA commented on HIVE-18221:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 6 new + 205
unchanged - 0 fixed = 211 total (was 205) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / fb85336 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8116/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: common standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8116/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> test acid default
> -
>
> Key: HIVE-18221
> URL: https://issues.apache.org/jira/browse/HIVE-18221
> Project: Hive
> Issue Type: Test
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch,
> HIVE-18221.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17333) Schema changes in HIVE-12274 for Oracle may not work for upgrade
[
https://issues.apache.org/jira/browse/HIVE-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-17333:
-
Attachment: HIVE-17333.1.patch
Revising the patch to remove the NOT NULL clauses in the COLUMN_NAME changes.
This wasnt tested before.
> Schema changes in HIVE-12274 for Oracle may not work for upgrade
>
>
> Key: HIVE-17333
> URL: https://issues.apache.org/jira/browse/HIVE-17333
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-17333.1.patch, HIVE-17333.patch
>
>
> According to
> https://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:1770086700346491686
> (reported in HIVE-12274)
> The alter table command to change the column datatype from {{VARCHAR}} to
> {{CLOB}} may not work. So the correct way to accomplish this is to add a new
> temp column, copy the value from the current column, drop the current column
> and rename the new column to old column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17333) Schema changes in HIVE-12274 for Oracle may not work for upgrade
[
https://issues.apache.org/jira/browse/HIVE-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-17333:
-
Status: Patch Available (was: In Progress)
> Schema changes in HIVE-12274 for Oracle may not work for upgrade
>
>
> Key: HIVE-17333
> URL: https://issues.apache.org/jira/browse/HIVE-17333
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-17333.1.patch, HIVE-17333.patch
>
>
> According to
> https://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:1770086700346491686
> (reported in HIVE-12274)
> The alter table command to change the column datatype from {{VARCHAR}} to
> {{CLOB}} may not work. So the correct way to accomplish this is to add a new
> temp column, copy the value from the current column, drop the current column
> and rename the new column to old column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-17333) Schema changes in HIVE-12274 for Oracle may not work for upgrade
[
https://issues.apache.org/jira/browse/HIVE-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-17333 started by Naveen Gangam.
> Schema changes in HIVE-12274 for Oracle may not work for upgrade
>
>
> Key: HIVE-17333
> URL: https://issues.apache.org/jira/browse/HIVE-17333
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-17333.1.patch, HIVE-17333.patch
>
>
> According to
> https://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:1770086700346491686
> (reported in HIVE-12274)
> The alter table command to change the column datatype from {{VARCHAR}} to
> {{CLOB}} may not work. So the correct way to accomplish this is to add a new
> temp column, copy the value from the current column, drop the current column
> and rename the new column to old column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17333) Schema changes in HIVE-12274 for Oracle may not work for upgrade
[
https://issues.apache.org/jira/browse/HIVE-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-17333:
-
Status: Open (was: Patch Available)
> Schema changes in HIVE-12274 for Oracle may not work for upgrade
>
>
> Key: HIVE-17333
> URL: https://issues.apache.org/jira/browse/HIVE-17333
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-17333.patch
>
>
> According to
> https://asktom.oracle.com/pls/asktom/f?p=100:11:0P11_QUESTION_ID:1770086700346491686
> (reported in HIVE-12274)
> The alter table command to change the column datatype from {{VARCHAR}} to
> {{CLOB}} may not work. So the correct way to accomplish this is to add a new
> temp column, copy the value from the current column, drop the current column
> and rename the new column to old column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18079: --- Attachment: HIVE-18079.4.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18088) Add WM event traces at query level for debugging
[ https://issues.apache.org/jira/browse/HIVE-18088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279250#comment-16279250 ] Sergey Shelukhin commented on HIVE-18088: - +1 pending tests, one nit can be fixed on commit > Add WM event traces at query level for debugging > > > Key: HIVE-18088 > URL: https://issues.apache.org/jira/browse/HIVE-18088 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18088.1.patch, HIVE-18088.2.patch, > HIVE-18088.3.patch, HIVE-18088.4.patch, HIVE-18088.5.patch, > HIVE-18088.6.patch, HIVE-18088.WIP.patch > > > For debugging and testing purpose, expose workload manager events via /jmx > endpoint and print summary at the scope of query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18079: --- Attachment: (was: HIVE-18079.3.patch) > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18029) beeline - support proper usernames based on the URL arg
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279242#comment-16279242
]
Sergey Shelukhin commented on HIVE-18029:
-
Probably not, it's more of a bugfix.
> beeline - support proper usernames based on the URL arg
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18029.patch
>
>
> Update: looks like the argument connection URL is not handled consistently
> with the connect command one, and the latter passes on user name that's
> entered in the prompt correctly.
> So,
> {noformat}
> !connect (url) => prompt; the username on HS2 side is whatever is entered in
> the prompt
> beeline -u (url) => anonymous (no prompt)
> !connect (url);user=foo => foo
> beeline -u (url);user=foo => anonymous
> beeline -n foo -u (url with or without the user) => foo
> {noformat}
> I'm going to add support for extracting the user from the -u argument,
> similar to connect argument
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18189) Order by position does not work when cbo is disabled
[
https://issues.apache.org/jira/browse/HIVE-18189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279230#comment-16279230
]
Daniel Dai commented on HIVE-18189:
---
named_column_join.q and named_column_join.q pass locally. Tests are clear.
> Order by position does not work when cbo is disabled
>
>
> Key: HIVE-18189
> URL: https://issues.apache.org/jira/browse/HIVE-18189
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-18189.1.patch, HIVE-18189.2.patch,
> HIVE-18189.3.patch
>
>
> Investigating a failed query:
> {code}
> set hive.cbo.enable=false;
> set hive.orderby.position.alias=true;
> select distinct age from student order by 1 desc limit 20;
> {code}
> The query does not sort the output correctly when cbo is
> disabled/inactivated. I found two issues:
> 1. "order by position" query is broken by HIVE-16774
> 2. In particular, select distinct query never work for "order by position"
> query
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18224) Introduce interface above driver
[
https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279225#comment-16279225
]
Hive QA commented on HIVE-18224:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900704/HIVE-18224.01.patch
{color:green}SUCCESS:{color} +1 due to 40 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 3 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[jira] [Updated] (HIVE-18220) Workload Management tables have broken constraints defined on postgres schema
[
https://issues.apache.org/jira/browse/HIVE-18220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18220:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the patch!
> Workload Management tables have broken constraints defined on postgres schema
> -
>
> Key: HIVE-18220
> URL: https://issues.apache.org/jira/browse/HIVE-18220
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-18220.patch
>
>
> Schema initialization on Postgres fails with the following error:
> {noformat}
> 0: jdbc:postgresql://localhost.localdomain:54> ALTER TABLE ONLY "WM_POOL" ADD
> CO
> NSTRAINT "UNIQUE_WM_RESOURCEPLAN" UNIQUE ("NAME")
> Error: ERROR: column "NAME" named in key does not exist (state=42703,code=0)
> Closing: 0: jdbc:postgresql://localhost.localdomain:5432/hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent !!
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:559)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1183)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:957)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:935)
> at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:582)
> ... 8 more
> {noformat}
> It is due to couple on incorrect constraint definitions in the schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18153) refactor reopen and file management in TezTask
[ https://issues.apache.org/jira/browse/HIVE-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18153: Attachment: HIVE-18153.01.patch > refactor reopen and file management in TezTask > -- > > Key: HIVE-18153 > URL: https://issues.apache.org/jira/browse/HIVE-18153 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18153.01.patch, HIVE-18153.patch > > > TezTask reopen relies on getting the same session object in terms of setup; > WM reopen returns a new session from the pool. > The former has the advantage of not having to reupload files and stuff... but > the object reuse results in a lot of ugly code, and also reopen might be > slower on average with the session pool than just getting a session from the > pool. Either WM needs to do the object-preserving reopen, or TezTask needs to > be refactored. It looks like DAG would have to be rebuilt to do the latter > because of some paths tied to a directory of the old session. Let me see if I > can get around that; if not we can do the former; and then if the former > results in too much ugly code in WM to account for object reuse for different > Tez client I'd do the latter anyway since it's a failure path :) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18224) Introduce interface above driver
[
https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279154#comment-16279154
]
Hive QA commented on HIVE-18224:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 15 new + 216 unchanged - 4
fixed = 231 total (was 220) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} service: The patch generated 1 new + 60 unchanged - 1
fixed = 61 total (was 61) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/core: The patch generated 2 new + 5 unchanged
- 1 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / f631241 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8115/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8115/yetus/diff-checkstyle-service.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8115/yetus/diff-checkstyle-hcatalog_core.txt
|
| modules | C: ql service cli hcatalog/core hcatalog/hcatalog-pig-adapter
hcatalog/server-extensions hcatalog/webhcat/java-client hcatalog/streaming
itests/hcatalog-unit itests/hive-unit itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8115/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce interface above driver
>
>
> Key: HIVE-18224
> URL: https://issues.apache.org/jira/browse/HIVE-18224
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18224.01.patch
>
>
> Add an interface above driver; and use it outside of ql.
> The goal is to enable the overlaying of the Driver with some strategy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-15393) Update Guava version
[ https://issues.apache.org/jira/browse/HIVE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279128#comment-16279128 ] Jesus Camacho Rodriguez commented on HIVE-15393: [~bslim], some of the tests need update? Any reason to not upgrade directly to guava 21 (3.0 seems like a good moment to do this)? > Update Guava version > > > Key: HIVE-15393 > URL: https://issues.apache.org/jira/browse/HIVE-15393 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-15393.2.patch, HIVE-15393.patch > > > Druid base code is using newer version of guava 16.0.1 that is not compatible > with the current version used by Hive. > FYI Hadoop project is moving to Guava 18 not sure if it is better to move to > guava 18 or even 19. > https://issues.apache.org/jira/browse/HADOOP-10101 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279121#comment-16279121
]
Hive QA commented on HIVE-15939:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900702/HIVE-15939.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8114/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8114/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8114/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-8114/succeeded/238_UTBatch_itests__hive-unit_8_tests,
remoteFile=/home/hiveptest/35.225.218.206-hiveptest-0/logs/,
getExitCode()=255, getException()=null, getUser()=hiveptest,
getHost()=35.225.218.206, getInstance()=0]: 'ssh: connect to host
35.225.218.206 port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 35.225.218.206 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 35.225.218.206 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 35.225.218.206 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 35.225.218.206 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900702 - PreCommit-HIVE-Build
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18185) update insert_values_orig_table_use_metadata.q.out
[ https://issues.apache.org/jira/browse/HIVE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279117#comment-16279117 ] Zoltan Haindrich commented on HIVE-18185: - +1 > update insert_values_orig_table_use_metadata.q.out > -- > > Key: HIVE-18185 > URL: https://issues.apache.org/jira/browse/HIVE-18185 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Bertalan Kondrat > Attachments: HIVE-18185.patch > > > some time back this test was skipped; and put back in: HIVE-17076 > however...possibly some other changes were also present around that time; > which have caused a 1 byte difference in totalSize; and after that some stats > changes came by (HIVE-16827 / others)? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18185) update insert_values_orig_table_use_metadata.q.out
[ https://issues.apache.org/jira/browse/HIVE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bertalan Kondrat updated HIVE-18185: Status: Patch Available (was: Open) > update insert_values_orig_table_use_metadata.q.out > -- > > Key: HIVE-18185 > URL: https://issues.apache.org/jira/browse/HIVE-18185 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Bertalan Kondrat > Attachments: HIVE-18185.patch > > > some time back this test was skipped; and put back in: HIVE-17076 > however...possibly some other changes were also present around that time; > which have caused a 1 byte difference in totalSize; and after that some stats > changes came by (HIVE-16827 / others)? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18185) update insert_values_orig_table_use_metadata.q.out
[ https://issues.apache.org/jira/browse/HIVE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bertalan Kondrat updated HIVE-18185: Attachment: HIVE-18185.patch > update insert_values_orig_table_use_metadata.q.out > -- > > Key: HIVE-18185 > URL: https://issues.apache.org/jira/browse/HIVE-18185 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Bertalan Kondrat > Attachments: HIVE-18185.patch > > > some time back this test was skipped; and put back in: HIVE-17076 > however...possibly some other changes were also present around that time; > which have caused a 1 byte difference in totalSize; and after that some stats > changes came by (HIVE-16827 / others)? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18149) Stats: rownum estimation from datasize underestimates in most cases
[ https://issues.apache.org/jira/browse/HIVE-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18149: Attachment: HIVE-18149.01.patch > Stats: rownum estimation from datasize underestimates in most cases > --- > > Key: HIVE-18149 > URL: https://issues.apache.org/jira/browse/HIVE-18149 > Project: Hive > Issue Type: Sub-task > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18149.01.patch, HIVE-18149.01wip01.patch > > > rownum estimation is based on the following fact as of now: > * datasize being used from the following sources: > ** basicstats aggregates the loaded "on-heap" row sizes ; other readers are > able to give "raw size" estimation - I've checked orc; but I'm sure others > will do the sameapi docs are a bit vague about the methods purpose... > ** if the basicstats level info is not available; the filesystem level > "file-size-sums" are used as the "raw data size" ; which is multiplied by the > [deserialization > ratio|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L261] > ; which is currently 1. > the problem with all of this is that deser factor is 1; and that rowsize > counts in the online object headers.. > example; 20 rows are loaded into a partition > [columnstats_partlvl_dp.q|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L7] > after HIVE-18108 [this > explain|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/test/queries/clientpositive/columnstats_partlvl_dp.q#L25] > will estimate the rowsize of the table to be 404 bytes; however the 20 rows > of text is only 169 bytes...so it ends up with 0 rows... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279037#comment-16279037
]
Hive QA commented on HIVE-15939:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} serde: The patch generated 0 new + 192 unchanged - 7
fixed = 192 total (was 199) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 444 unchanged - 48
fixed = 444 total (was 492) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / f631241 |
| Default Java | 1.8.0_111 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8114/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279002#comment-16279002
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900714/HIVE-18079.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8113/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8113/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8113/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-12-05 18:26:15.911
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8113/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-12-05 18:26:15.914
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f631241 HIVE-18005: Improve size estimation for array() to be
not 0 (Zoltan Haindrich, reviewed by Vineet Garg)
+ git clean -f -d
Removing
standalone-metastore/src/test/java/org/apache/hadoop/hive/common/ndv/hll/TestHyperLogLogMerge.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f631241 HIVE-18005: Improve size estimation for array() to be
not 0 (Zoltan Haindrich, reviewed by Vineet Garg)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-12-05 18:26:18.324
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8113/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: data/conf/hive-site.xml:319
error: data/conf/hive-site.xml: patch does not apply
error: patch failed: data/conf/llap/hive-site.xml:343
error: data/conf/llap/hive-site.xml: patch does not apply
error: patch failed: data/conf/spark/yarn-client/hive-site.xml:295
error: data/conf/spark/yarn-client/hive-site.xml: patch does not apply
error: patch failed: data/conf/tez/hive-site.xml:288
error: data/conf/tez/hive-site.xml: patch does not apply
error: patch failed:
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out:184
error:
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out:
patch does not apply
error: patch failed:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreAlterColumnPar.java:18
error:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreAlterColumnPar.java:
patch does not apply
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Context.java:509
error: ql/src/java/org/apache/hadoop/hive/ql/Context.java: patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:1993
error: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6827
error: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: patch
does not apply
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java:52
error: ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java: patch does
not apply
error: patch failed: testutils/ptest2/src/main/resources/smart-apply-patch.sh:20
error: testutils/ptest2/src/main/resources/smart-apply-patch.sh: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900714 -
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278999#comment-16278999
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12900572/HIVE-18079.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 11537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llapdecider]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
(batchId=231)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8112/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8112/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8112/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12900572 - PreCommit-HIVE-Build
> Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator
> bit-size
>
>
> Key: HIVE-18079
> URL: https://issues.apache.org/jira/browse/HIVE-18079
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Statistics
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch,
> HIVE-18079.3.patch
>
>
> HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its
> mathematical hash distribution & construction.
> Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second
> scan over the data-set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Attachment: HIVE-18221.03.patch > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18224) Introduce interface above driver
[ https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18224: --- Assignee: Zoltan Haindrich > Introduce interface above driver > > > Key: HIVE-18224 > URL: https://issues.apache.org/jira/browse/HIVE-18224 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18224.01.patch > > > Add an interface above driver; and use it outside of ql. > The goal is to enable the overlaying of the Driver with some strategy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18225) disaggreement in results; vector_decimal_mapjoin.q
[
https://issues.apache.org/jira/browse/HIVE-18225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278931#comment-16278931
]
Zoltan Haindrich commented on HIVE-18225:
-
[~sershe], [~teddy.choi]: I'm not sure wheteher this is connect to the other
issue; but I've to say that I see the llap_fast related result diff almost all
the time: when {{hive.stats.column.autogather}} to true...this might be
possibly a more stable result difference...
> disaggreement in results; vector_decimal_mapjoin.q
> --
>
> Key: HIVE-18225
> URL: https://issues.apache.org/jira/browse/HIVE-18225
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>
> one of these out files have changed for me...and it turned out that it's not
> clear which output should be the expected; since we have different version
> recorded :D
> for the last query in {{vector_decimal_mapjoin.q}} ; spark and clidriver
> gives 1 result; in the meanwhile llap returns a whole bunch of results.
> {code}
> $ tail `find ql -name vector_decimal_mapjoin.q.out`
> ==> ql/src/test/results/clientpositive/vector_decimal_mapjoin.q.out <==
> PREHOOK: type: QUERY
> PREHOOK: Input: default@t1_small
> PREHOOK: Input: default@t2_small
> A masked pattern was here
> POSTHOOK: query: select t1_small.`dec`, t1_small.value_dec, t2_small.`dec`,
> t2_small.value_dec from t1_small join t2_small on
> (t1_small.`dec`=t2_small.`dec`)
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@t1_small
> POSTHOOK: Input: default@t2_small
> A masked pattern was here
> 89.00 15.09 89 15
> ==> ql/src/test/results/clientpositive/llap/vector_decimal_mapjoin.q.out <==
> 9.00 48.96 9 34
> 9.00 48.96 9 38
> 9.00 48.96 9 41
> 9.00 48.96 9 42
> 9.00 48.96 9 45
> 9.00 48.96 9 48
> 9.00 48.96 9 49
> 9.00 48.96 9 5
> 9.00 48.96 9 7
> 9.00 48.96 9 7
> ==> ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out <==
> PREHOOK: type: QUERY
> PREHOOK: Input: default@t1_small
> PREHOOK: Input: default@t2_small
> A masked pattern was here
> POSTHOOK: query: select t1_small.`dec`, t1_small.value_dec, t2_small.`dec`,
> t2_small.value_dec from t1_small join t2_small on
> (t1_small.`dec`=t2_small.`dec`)
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@t1_small
> POSTHOOK: Input: default@t2_small
> A masked pattern was here
> 89.00 15.09 89 15
> {code}
> I think the TestClidriver based one can be "changed" to the other result; by
> setting {{hive.stats.column.autogather=true}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18079: --- Attachment: HIVE-18079.3.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-18079.1.patch, HIVE-18079.2.patch, > HIVE-18079.3.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18036) Stats: Remove usage of clone() methods
[
https://issues.apache.org/jira/browse/HIVE-18036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278856#comment-16278856
]
Bertalan Kondrat commented on HIVE-18036:
-
Thank you for help and for the review [~kgyrtkirk]!
> Stats: Remove usage of clone() methods
> --
>
> Key: HIVE-18036
> URL: https://issues.apache.org/jira/browse/HIVE-18036
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Bertalan Kondrat
> Fix For: 3.0.0
>
> Attachments: HIVE-18036.2.patch, HIVE-18036.3.patch, HIVE-18036.patch
>
>
> {{Statistics}} and {{ColStats}} implements cloneable; however they never
> throw clonenotsupported; and this causes try / catch blocks which are just
> noise
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18224) Introduce interface above driver
[ https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18224: Status: Patch Available (was: Open) > Introduce interface above driver > > > Key: HIVE-18224 > URL: https://issues.apache.org/jira/browse/HIVE-18224 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Zoltan Haindrich > Attachments: HIVE-18224.01.patch > > > Add an interface above driver; and use it outside of ql. > The goal is to enable the overlaying of the Driver with some strategy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18224) Introduce interface above driver
[ https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18224: Attachment: HIVE-18224.01.patch #1) * I hope everything will be in order...this patch should not impose any changes > Introduce interface above driver > > > Key: HIVE-18224 > URL: https://issues.apache.org/jira/browse/HIVE-18224 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Zoltan Haindrich > Attachments: HIVE-18224.01.patch > > > Add an interface above driver; and use it outside of ql. > The goal is to enable the overlaying of the Driver with some strategy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15939) Make cast expressions comply more to sql2011
[
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-15939:
Attachment: HIVE-15939.07.patch
#7)
* q.out fixes; fix some minor issues
> Make cast expressions comply more to sql2011
>
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15939.07.patch, HIVE-15939.1.patch,
> HIVE-15939.2.patch, HIVE-15939.3.patch, HIVE-15939.4.patch,
> HIVE-15939.5.patch, HIVE-15939.6.patch
>
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
