[jira] [Issue Comment Deleted] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20644: Comment: was deleted (was: +1, pending tests) > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[
https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637832#comment-16637832
]
Hive QA commented on HIVE-20531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942248/HIVE-20531.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15011 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14217/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14217/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942248 - PreCommit-HIVE-Build
> Repl load on cloud storage file system can skip redundant move or add
> partition tasks.
> --
>
> Key: HIVE-20531
> URL: https://issues.apache.org/jira/browse/HIVE-20531
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch,
> HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch,
> HIVE-20531.06.patch, HIVE-20531.07.patch, HIVE-20531.08.patch
>
>
> In replication load, both add partition and insert operations are handled
> through import. Import creates 3 major tasks. Copy, add partition and move.
> Copy does the copy of data from source location to staging directory. Then
> add partition (which runs in parallel to copy) creates the partition in meta
> store. Its a no op in case of insert and by the time this ddl task is
> executed for insert partition would be already present. The third operation
> is move. Which actually moves the file from staging directory to actual
> location. And then in case of insert it adds the insert event to notification
> table. It does this for add partition operation which is redundant as the
> event for add partition would have been written already by ddl task. With the
> optimization to copy directly to actual table location in S3, move task can
> be avoided for add partition operation replay and replay of insert need not
> create the add partition (ddl) task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20674:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Patch pushed to master. Thanks Vaibhav for review!
> TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
> ---
>
> Key: HIVE-20674
> URL: https://issues.apache.org/jira/browse/HIVE-20674
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20674.1.patch
>
>
> Intermittent but frequently:
> {code}
> java.sql.SQLException: org.apache.thrift.transport.TTransportException
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
>
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637824#comment-16637824 ] anishek commented on HIVE-20679: [~diser555] can you please attach a sample here in with that scale ? > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitificaion, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[
https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637818#comment-16637818
]
Hive QA commented on HIVE-20531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 680
unchanged - 0 fixed = 683 total (was 680) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 144 unchanged - 1
fixed = 145 total (was 145) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14217/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14217/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14217/yetus/diff-checkstyle-ql.txt
|
| modules | C: common itests/hive-unit ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14217/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl load on cloud storage file system can skip redundant move or add
> partition tasks.
> --
>
> Key: HIVE-20531
> URL: https://issues.apache.org/jira/browse/HIVE-20531
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637806#comment-16637806
]
Hive QA commented on HIVE-20610:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942288/HIVE-20610.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14995 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14215/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14215/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14215/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942288 - PreCommit-HIVE-Build
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
>
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637807#comment-16637807
]
Hive QA commented on HIVE-20646:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942231/HIVE-20646.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942231/HIVE-20646.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942231 - PreCommit-HIVE-Build
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Metastore, Performance, pull-request-available
> Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch
>
>
> If the partition filter condition has "is not null" then the filter query
> isn't getting pushed to the SQL query in RDMBS.
> This slows down metastore api calls for getting list of partitions with
> filter condition.
> This condition gets added by optimizer in many cases so this is affecting
> many queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20680) hive bootstrap missing partitions in replicated db
[ https://issues.apache.org/jira/browse/HIVE-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-20680: -- Labels: pull-request-available (was: ) > hive bootstrap missing partitions in replicated db > -- > > Key: HIVE-20680 > URL: https://issues.apache.org/jira/browse/HIVE-20680 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > The issue is > 1. When bootstrap was going on, kill query was called to kill the repl load > command. > 2. During restart, one table with no partition set the scope to table as the > ckpt property was not yet set for that table. > 3. Due to this, all partitioned table after this didn't get their tasks > related to partition appended in the root task if the ckpt property is > already updated for those tables. > The fix is to reset the table scope to false if for that table there are no > tasks added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20680) hive bootstrap missing partitions in replicated db
[ https://issues.apache.org/jira/browse/HIVE-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637772#comment-16637772 ] ASF GitHub Bot commented on HIVE-20680: --- GitHub user maheshk114 opened a pull request: https://github.com/apache/hive/pull/443 HIVE-20680 : hive bootstrap missing partitions in replicated db You can merge this pull request into a Git repository by running: $ git pull https://github.com/maheshk114/hive HIVE-20680 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/443.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #443 commit 114b9cb8df9e23f0155364fe1bec143f45817318 Author: Mahesh Kumar Behera Date: 2018-10-03T08:36:02Z HIVE-20680 : hive bootstrap missing partitions in replicated db > hive bootstrap missing partitions in replicated db > -- > > Key: HIVE-20680 > URL: https://issues.apache.org/jira/browse/HIVE-20680 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > The issue is > 1. When bootstrap was going on, kill query was called to kill the repl load > command. > 2. During restart, one table with no partition set the scope to table as the > ckpt property was not yet set for that table. > 3. Due to this, all partitioned table after this didn't get their tasks > related to partition appended in the root task if the ckpt property is > already updated for those tables. > The fix is to reset the table scope to false if for that table there are no > tasks added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637757#comment-16637757
]
Hive QA commented on HIVE-20610:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14215/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| modules | C: itests/hcatalog-unit U: itests/hcatalog-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14215/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
>
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637751#comment-16637751
]
Hive QA commented on HIVE-20647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942247/HIVE-20647.8.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=154)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14214/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14214/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942247 - PreCommit-HIVE-Build
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch, HIVE-20647.8.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field
> method called from constructor of superclass)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637730#comment-16637730
]
Hive QA commented on HIVE-20647:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} itests/util: The patch generated 0 new + 148
unchanged - 17 fixed = 148 total (was 165) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} itests/util generated 0 new + 48 unchanged - 4 fixed
= 48 total (was 52) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14214/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util itests/hive-unit U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14214/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch, HIVE-20647.8.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR:
[jira] [Comment Edited] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637723#comment-16637723 ] Jesus Camacho Rodriguez edited comment on HIVE-20657 at 10/4/18 2:12 AM: - The clean run of the other one is 1 month old on the other hand (HIVE-20338). Maybe it has to do with that one... I will check as soon as possible, but I would recommend reverting since this is ruining all ptest runs. was (Author: jcamachorodriguez): The clean run of the other one is 1 month old on the other hand (HIVE-20338). Maybe it has to do with that one... I will check as soon as possible. > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637723#comment-16637723 ] Jesus Camacho Rodriguez commented on HIVE-20657: The clean run of the other one is 1 month old on the other hand (HIVE-20338). Maybe it has to do with that one... I will check as soon as possible. > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[
https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637718#comment-16637718
]
Jesus Camacho Rodriguez commented on HIVE-20657:
{{org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]}} has
been failing consistently for the last 4 runs. Can we check whether this is the
offending commit? (There are only 2 commits within the last 10 hours and the
other one got a green run).
> pre-allocate LLAP cache at init time
>
>
> Key: HIVE-20657
> URL: https://issues.apache.org/jira/browse/HIVE-20657
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20657.01.patch, HIVE-20657.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637714#comment-16637714
]
Hive QA commented on HIVE-20652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942336/HIVE-20652.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15011 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=154)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.tablesGetExists
(batchId=220)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14213/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14213/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942336 - PreCommit-HIVE-Build
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.patch,
> external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637687#comment-16637687
]
Jaume M commented on HIVE-20671:
[~akolb] they seem to be running fine in master so it must be something
specific of this patch.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-20563:
Attachment: HIVE-20563.08.patch
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:136)
>
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637679#comment-16637679
]
Hive QA commented on HIVE-20652:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc-handler: The patch generated 1 new + 32 unchanged
- 2 fixed = 33 total (was 34) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 176 unchanged - 0
fixed = 181 total (was 176) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14213/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14213/yetus/diff-checkstyle-jdbc-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14213/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14213/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests jdbc-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14213/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch,
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637658#comment-16637658
]
Hive QA commented on HIVE-20623:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942234/HIVE-20623.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=154)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14212/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14212/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942234 - PreCommit-HIVE-Build
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.04.patch, HIVE-20623.patch,
> hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales, customer
> where customer_sk = c_customer_sk
> group by c_customer_id
> order by total_price desc
> limit 100;
> {code}
> The hashtable used for all 3 joins are identical, which is loaded 3x times in
> the same LLAP instance because they are named.
> {code}
> cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
> {code}
> in the cache.
> If those are identical in nature (i.e vectorization, hashtable type etc),
> then the duplication is just wasted CPU, memory and network - using the cache
> name for hashtables which will be identical in layout would be extremely
> useful.
> In cases where the join is pushed through a UNION, those are identical.
> This optimization can only be done without concern for accidental delays when
> the same upstream task is generating all of these hashtables, which is what
> is achieved by the shared scan optimizer already.
> In case the shared work is not present, this has potential downsides - in
> case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1
> builder will block the other task from reading from Map 2, even though Map 2
> might have started after, but finished ahead of Map 1.
> So this specific optimization can always be considered for cases where the
> shared work unifies the operator tree and the parents of all the RS entries
> involved are same (& the RS layout is the same).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637639#comment-16637639 ] Vihang Karajgaonkar edited comment on HIVE-20306 at 10/4/18 12:07 AM: -- v18 patch fixes related issues found by findbugs was (Author: vihangk1): Fixed related issues found by findbugs > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637639#comment-16637639 ] Vihang Karajgaonkar commented on HIVE-20306: Fixed related issues found by findbugs > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: HIVE-20306.18.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[
https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637638#comment-16637638
]
Hive QA commented on HIVE-20623:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14212/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14212/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shared work: Extend sharing of map-join cache entries in LLAP
> -
>
> Key: HIVE-20623
> URL: https://issues.apache.org/jira/browse/HIVE-20623
> Project: Hive
> Issue Type: Improvement
> Components: llap, Logical Optimizer
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch,
> HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch,
> HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.04.patch,
> HIVE-20623.04.patch, HIVE-20623.04.patch, HIVE-20623.patch,
> hash-shared-work.json.txt, hash-shared-work.svg
>
>
> For a query like this
> {code}
> with all_sales as (
> select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt
> as ext_price from store_sales
> UNION ALL
> select ws_bill_customer_sk as customer_sk,
> ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
> UNION ALL
> select cs_bill_customer_sk as customer_sk, cs_ext_sales_price -
> cs_ext_discount_amt as ext_price from catalog_sales)
> select sum(ext_price) total_price, c_customer_id from all_sales,
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637625#comment-16637625
]
Jesus Camacho Rodriguez commented on HIVE-20652:
[~ashutoshc], [~daijy], could you take a look?
https://reviews.apache.org/r/68922/
Thanks
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.patch,
> external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: HIVE-20652.01.patch
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.patch,
> external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
>
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637612#comment-16637612
]
Hive QA commented on HIVE-20644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942241/HIVE-20644.03
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14211/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942241/HIVE-20644.03 was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942241 - PreCommit-HIVE-Build
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some sensitive information is leaked to a user through exception
> message. That information may not be available to the user otherwise. Hence
> it's a kind of security breach or violation of access control.
> The contents of the row or the arguments to a function may be useful for
> debugging and hence it's worth to add those to logs. Hence proposal here to
> log a separate message with log level DEBUG or INFO containing the string
> representation of the row. Users can configure their logging so that
> DEBUG/INFO messages do not go to the client but at the same time are
> available in the hive server logs for debugging. The actual exception message
> will not contain any sensitive data like row data or argument data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637610#comment-16637610
]
Hive QA commented on HIVE-20644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942241/HIVE-20644.03
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15005 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=194)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams]
(batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=154)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942241 - PreCommit-HIVE-Build
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some sensitive information is leaked to a user through exception
> message. That information may not be available to the user otherwise. Hence
> it's a kind of security breach or violation of access control.
> The contents of the row or the arguments to a function may be useful for
> debugging and hence it's worth to add those to logs. Hence proposal here to
> log a separate message with log level DEBUG or INFO containing the string
> representation of the row. Users can configure their logging so that
> DEBUG/INFO messages do not go to the client but at the same time are
> available in the hive server logs for debugging. The actual exception message
> will not contain any sensitive data like row data or argument data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637584#comment-16637584
]
Hive QA commented on HIVE-20644:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14210/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14210/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some sensitive information is leaked to a user through exception
> message. That information may not be available to the user otherwise. Hence
> it's a kind of security breach or violation of access control.
> The contents of the row or the arguments to a function
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637557#comment-16637557
]
Hive QA commented on HIVE-20646:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942231/HIVE-20646.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14209/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14209/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14209/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942231 - PreCommit-HIVE-Build
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Metastore, Performance, pull-request-available
> Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch
>
>
> If the partition filter condition has "is not null" then the filter query
> isn't getting pushed to the SQL query in RDMBS.
> This slows down metastore api calls for getting list of partitions with
> filter condition.
> This condition gets added by optimizer in many cases so this is affecting
> many queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637548#comment-16637548 ] Ashutosh Chauhan commented on HIVE-20686: - +1 pending tests. > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637542#comment-16637542
]
Hive QA commented on HIVE-20646:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 52 unchanged - 1
fixed = 52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14209/dev-support/hive-personality.sh
|
| git revision | master / 240bfb4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common ql
standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14209/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels:
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637529#comment-16637529 ] Eugene Ilchenko commented on HIVE-20679: Entries for tables with deep Avro schemas can get even larger. We have seen 500K entries for a single added partition. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitificaion, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637496#comment-16637496
]
Hive QA commented on HIVE-20652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942232/HIVE-20652.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15011 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_handler]
(batchId=173)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14208/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14208/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942232 - PreCommit-HIVE-Build
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Attachment: HIVE-12254.5.patch > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Status: Patch Available (was: In Progress) > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Attachment: (was: HIVE-12254.5.patch) > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-12254: Status: In Progress (was: Patch Available) > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637479#comment-16637479
]
Hive QA commented on HIVE-20652:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} jdbc-handler: The patch generated 1 new + 32 unchanged
- 2 fixed = 33 total (was 34) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 5 new + 176 unchanged - 0
fixed = 181 total (was 176) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14208/dev-support/hive-personality.sh
|
| git revision | master / a06a370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14208/yetus/diff-checkstyle-jdbc-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14208/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14208/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests jdbc-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14208/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.patch,
[jira] [Commented] (HIVE-20688) Update Committer List
[ https://issues.apache.org/jira/browse/HIVE-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637477#comment-16637477 ] Andrew Sherman commented on HIVE-20688: --- + LGTM > Update Committer List > - > > Key: HIVE-20688 > URL: https://issues.apache.org/jira/browse/HIVE-20688 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Minor > Attachments: HIVE-20688.1.patch > > > Please update committer list: > Name: Janaki Lahorani > Apache ID: janaki > Organization: Cloudera -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20688) Update Committer List
[ https://issues.apache.org/jira/browse/HIVE-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20688: --- Attachment: HIVE-20688.1.patch > Update Committer List > - > > Key: HIVE-20688 > URL: https://issues.apache.org/jira/browse/HIVE-20688 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Minor > Attachments: HIVE-20688.1.patch > > > Please update committer list: > Name: Janaki Lahorani > Apache ID: janaki > Organization: Cloudera -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20688) Update Committer List
[ https://issues.apache.org/jira/browse/HIVE-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-20688: --- Status: Patch Available (was: Open) > Update Committer List > - > > Key: HIVE-20688 > URL: https://issues.apache.org/jira/browse/HIVE-20688 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Minor > > Please update committer list: > Name: Janaki Lahorani > Apache ID: janaki > Organization: Cloudera -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20688) Update Committer List
[ https://issues.apache.org/jira/browse/HIVE-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani reassigned HIVE-20688: -- > Update Committer List > - > > Key: HIVE-20688 > URL: https://issues.apache.org/jira/browse/HIVE-20688 > Project: Hive > Issue Type: Bug >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Minor > > Please update committer list: > Name: Janaki Lahorani > Apache ID: janaki > Organization: Cloudera -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637463#comment-16637463 ] Thejas M Nair commented on HIVE-20644: -- For completeness, maybe make similar update in ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java as well ? Also, DEBUG level is often used for debugging variety of problems, and people can set entire HS2 logging at that level. Sometimes production systems can be put in DEBUG level logging for sometime while troubleshooting an issue. AFAIK, debug level messages in HS2 would be sent to beeline if HS2 is running in debug mode and user has set hive.server2.logging.operation.level=verbose. (Since this error is happening in the tasks, its possible that this doesn't get sent currently. But again, that can change in future) I think its unusual to have production systems running with TRACE level logs, so maybe log the rowString part only in trace level logging ? I would expect only specific classes to be enabled at trace level logging, so that would be a safer option IMO. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20657: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[
https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637445#comment-16637445
]
Hive QA commented on HIVE-17043:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942306/HIVE-17043.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 15011 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join_pkfk]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_vc] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_4]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_sw]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_6]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_recursive_mapjoin]
(batchId=189)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_vc]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query17]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query22]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query24]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query25]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query29]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query32]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query45]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query57]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query65]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query66]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query67]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query72]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query85]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query91]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query92]
(batchId=267)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query99]
(batchId=267)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query17]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query22]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query24]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query25]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query29]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query32]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query45]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query57]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query65]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query67]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query70]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query72]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query85]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query91]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query92]
(batchId=265)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query99]
(batchId=265)
{noformat}
Test results:
[jira] [Resolved] (HIVE-20645) Vectorization: Implicit casting causes scratch vector reuse Wrong Results
[ https://issues.apache.org/jira/browse/HIVE-20645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline resolved HIVE-20645. - Resolution: Duplicate Folding this into HIVE-20563. > Vectorization: Implicit casting causes scratch vector reuse Wrong Results > - > > Key: HIVE-20645 > URL: https://issues.apache.org/jira/browse/HIVE-20645 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > The bug fix in HIVE-20563 exposes a Wrong Results bug in vectorized_cast.q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20657) pre-allocate LLAP cache at init time
[ https://issues.apache.org/jira/browse/HIVE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637434#comment-16637434 ] Sergey Shelukhin commented on HIVE-20657: - There's a test run with only that test failing, and the other tests from later runs don't repro locally... committing based on that run > pre-allocate LLAP cache at init time > > > Key: HIVE-20657 > URL: https://issues.apache.org/jira/browse/HIVE-20657 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20657.01.patch, HIVE-20657.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19291) Three underscores are in the CTAS example of the documentation
[
https://issues.apache.org/jira/browse/HIVE-19291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637426#comment-16637426
]
Nishant Bangarwa commented on HIVE-19291:
-
have fixed it in the wiki.
Thanks [~wsalazar] for reporting it.
> Three underscores are in the CTAS example of the documentation
> ---
>
> Key: HIVE-19291
> URL: https://issues.apache.org/jira/browse/HIVE-19291
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Reporter: Wade Salazar
>Assignee: Nishant Bangarwa
>Priority: Trivial
>
> on the page https://cwiki.apache.org/confluence/display/Hive/Druid+Integration
>
> {{The following example is provided}}
> {{"}}
> {{CREATE TABLE druid_table_1}}
> {{STORED BY }}{{'org.apache.hadoop.hive.druid.DruidStorageHandler'}}
> {{AS}}
> {{ `metric2`>;}}
> {{"}}
>
> {{There are 3 underscores in front of the time dimension where the code only
> executes if 2 underscores are provided}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19291) Three underscores are in the CTAS example of the documentation
[
https://issues.apache.org/jira/browse/HIVE-19291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa resolved HIVE-19291.
-
Resolution: Fixed
> Three underscores are in the CTAS example of the documentation
> ---
>
> Key: HIVE-19291
> URL: https://issues.apache.org/jira/browse/HIVE-19291
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Reporter: Wade Salazar
>Assignee: Nishant Bangarwa
>Priority: Trivial
>
> on the page https://cwiki.apache.org/confluence/display/Hive/Druid+Integration
>
> {{The following example is provided}}
> {{"}}
> {{CREATE TABLE druid_table_1}}
> {{STORED BY }}{{'org.apache.hadoop.hive.druid.DruidStorageHandler'}}
> {{AS}}
> {{ `metric2`>;}}
> {{"}}
>
> {{There are 3 underscores in front of the time dimension where the code only
> executes if 2 underscores are provided}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20687) Cancel Running Druid Query when a hive query is cancelled.
[
https://issues.apache.org/jira/browse/HIVE-20687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637419#comment-16637419
]
Nishant Bangarwa commented on HIVE-20687:
-
+cc [~ashutoshc] Any thoughts on this ? does this seems like a feasible
approach ?
> Cancel Running Druid Query when a hive query is cancelled.
> ---
>
> Key: HIVE-20687
> URL: https://issues.apache.org/jira/browse/HIVE-20687
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id
> is passed to druid.
> Druid also supports query cancellation by query id.
> Queries can be cancelled explicitly using their queryId by sending a DELETE
> request to following endpoint on the broker or router -
> {code}
> DELETE /druid/v2/{queryId}
> {code}
> Implementation wise we need to cancel a druid query when a RecordReader is
> closed.
> I believe Hive should already be calling close on open recordReaders when
> someone cancels a query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20687) Cancel Running Druid Query when a hive query is cancelled.
[
https://issues.apache.org/jira/browse/HIVE-20687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20687:
Description:
https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id is
passed to druid.
Druid also supports query cancellation by query id.
Queries can be cancelled explicitly using their queryId by sending a DELETE
request to following endpoint on the broker or router -
{code}
DELETE /druid/v2/{queryId}
{code}
Implementation wise we need to cancel a druid query when a RecordReader is
closed.
I believe Hive should already be calling close on open recordReaders when
someone cancels a query.
was:
https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id is
passed to druid.
Druid also supports query cancellation by query id.
Queries can be cancelled explicitly using their queryId by sending a DELETE
request to following endpoint on the broker or router -
{code}
DELETE /druid/v2/{queryId}
{code}
> Cancel Running Druid Query when a hive query is cancelled.
> ---
>
> Key: HIVE-20687
> URL: https://issues.apache.org/jira/browse/HIVE-20687
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id
> is passed to druid.
> Druid also supports query cancellation by query id.
> Queries can be cancelled explicitly using their queryId by sending a DELETE
> request to following endpoint on the broker or router -
> {code}
> DELETE /druid/v2/{queryId}
> {code}
> Implementation wise we need to cancel a druid query when a RecordReader is
> closed.
> I believe Hive should already be calling close on open recordReaders when
> someone cancels a query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20687) Cancel Running Druid Query when a hive query is cancelled.
[
https://issues.apache.org/jira/browse/HIVE-20687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa reassigned HIVE-20687:
---
> Cancel Running Druid Query when a hive query is cancelled.
> ---
>
> Key: HIVE-20687
> URL: https://issues.apache.org/jira/browse/HIVE-20687
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id
> is passed to druid.
> Druid also supports query cancellation by query id.
> Queries can be cancelled explicitly using their queryId by sending a DELETE
> request to following endpoint on the broker or router -
> {code}
> DELETE /druid/v2/{queryId}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637400#comment-16637400 ] Nishant Bangarwa commented on HIVE-20686: - +cc [~ashutoshc] A small patch to sync up query Ids between hive and druid to make it easy to trace hive query performance by exploring druid metrics based on query id. > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637398#comment-16637398 ] Sergey Shelukhin commented on HIVE-20672: - +1 I'm assuming the test failure is unrelated > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20672.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637397#comment-16637397
]
Vaibhav Gumashta commented on HIVE-20674:
-
+1
> TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
> ---
>
> Key: HIVE-20674
> URL: https://issues.apache.org/jira/browse/HIVE-20674
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20674.1.patch
>
>
> Intermittent but frequently:
> {code}
> java.sql.SQLException: org.apache.thrift.transport.TTransportException
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
>
[jira] [Updated] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20686: Status: Patch Available (was: Open) > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20686: Attachment: HIVE-20686.patch > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[
https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637389#comment-16637389
]
Hive QA commented on HIVE-17043:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 11 new + 51 unchanged - 3
fixed = 62 total (was 54) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14207/dev-support/hive-personality.sh
|
| git revision | master / a06a370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14207/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14207/yetus/whitespace-eol.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14207/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove non unique columns from group by keys if not referenced later
>
>
> Key: HIVE-17043
> URL: https://issues.apache.org/jira/browse/HIVE-17043
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch,
> HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch,
> HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch, HIVE-17043.9.patch
>
>
> Group by keys may be a mix of unique (or primary) keys and regular columns.
> In such cases presence of regular column won't alter cardinality of groups.
> So, if regular columns are not referenced later, they can be dropped from
> group by keys. Depending on operator tree may result in those columns not
> being read at all from disk in best case. In worst case, we will avoid
>
[jira] [Comment Edited] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637388#comment-16637388
]
Nishant Bangarwa edited comment on HIVE-20684 at 10/3/18 7:07 PM:
--
This one has multiple fixes to make analyze table column stats work for Druid -
# Add support for analyzing timestamp with timezone column type
# By default we were creating a druid scan query but were using a select query
reader leading to wrong parsing of results.
# When creating a default scan query, column list was not passed to druid. This
led to mismatch in column order between DruidSerde and results coming from
druid. Leading to classCastExceptions in some cases OR columns data getting
mixed.
# TableScanDesc was incorrectly passed virtual columns in list of columns for
non-native tables. Added a fix to only pass in VirtualColumns for native
tables.
+cc [~ashutoshc] [~jcamachorodriguez]
was (Author: nishantbangarwa):
This one has multiple fixes to make analyze table column stats work for Druid -
# Add support for analyzing timestamp with timezone column type
# By default we were creating a druid scan query but were using a select query
reader.
# When creating a default scan query, column list was not passed to druid. This
led to mismatch in column order between DruidSerde and results coming from
druid. Leading to classCastExceptions in some cases OR columns data getting
mixed.
# TableScanDesc was incorrectly passed virtual columns in list of columns for
non-native tables. Added a fix to only pass in VirtualColumns for native
tables.
+cc [~ashutoshc] [~jcamachorodriguez]
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637388#comment-16637388
]
Nishant Bangarwa commented on HIVE-20684:
-
This one has multiple fixes to make analyze table column stats work for Druid -
# Add support for analyzing timestamp with timezone column type
# By default we were creating a druid scan query but were using a select query
reader.
# When creating a default scan query, column list was not passed to druid. This
led to mismatch in column order between DruidSerde and results coming from
druid. Leading to classCastExceptions in some cases OR columns data getting
mixed.
# TableScanDesc was incorrectly passed virtual columns in list of columns for
non-native tables. Added a fix to only pass in VirtualColumns for native
tables.
+cc [~ashutoshc] [~jcamachorodriguez]
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20684:
Status: Patch Available (was: Open)
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19844) Make CSV SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637386#comment-16637386
]
Naveen Gangam commented on HIVE-19844:
--
[~achinn1] Are you actively working on this? if not, may I re-assign this so we
could bring this to a resolution? If you intend to work on it, could you please
provide a timeframe when you might be able to get to this? Thanks
> Make CSV SerDe First-Class SerDe
>
>
> Key: HIVE-19844
> URL: https://issues.apache.org/jira/browse/HIVE-19844
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Serializers/Deserializers
>Affects Versions: 3.0.0, 2.3.2, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: anand
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/CSV+Serde], there are
> some extras steps involved in getting the CSV SerDe working with Hive.
> {code}
> CREATE TABLE my_table(a string, b string, ...)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
> WITH SERDEPROPERTIES (
>"separatorChar" = "\t",
>"quoteChar" = "'",
>"escapeChar"= "\\"
> )
> STORED AS TEXTFILE;
> {code}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS TEXT_CSV}}
> {{STORED AS TEXT_TSV}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20684:
Attachment: HIVE-20684.patch
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[
https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637355#comment-16637355
]
Hive QA commented on HIVE-20676:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942219/HIVE-20676.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15010 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14206/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14206/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14206/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942219 - PreCommit-HIVE-Build
> HiveServer2: PrivilegeSynchronizer is not set to daemon status
> --
>
> Key: HIVE-20676
> URL: https://issues.apache.org/jira/browse/HIVE-20676
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20676.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Status: Patch Available (was: Open) > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch, HIVE-17043.9.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Status: Open (was: Patch Available) > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch, HIVE-17043.9.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[ https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17043: --- Attachment: HIVE-17043.9.patch > Remove non unique columns from group by keys if not referenced later > > > Key: HIVE-17043 > URL: https://issues.apache.org/jira/browse/HIVE-17043 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch, > HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch, > HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch, HIVE-17043.9.patch > > > Group by keys may be a mix of unique (or primary) keys and regular columns. > In such cases presence of regular column won't alter cardinality of groups. > So, if regular columns are not referenced later, they can be dropped from > group by keys. Depending on operator tree may result in those columns not > being read at all from disk in best case. In worst case, we will avoid > shuffling and sorting regular columns from mapper to reducer, which still > could be substantial CPU and network savings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages
[
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637349#comment-16637349
]
Daniel Dai commented on HIVE-20608:
---
Sounds like you shall write a .q test for that. Does it reproducible by setting
"hive.security.authorization.enabled"=true? It will enable the parameter
blacklist check.
> Incorrect handling of sql command args in hive service leading to misleading
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
> Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says:
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: Cannot modify set hive.exec.dynamic.partition at
> runtime. It is not in list of params that are allowed to be modified at
> runtime
> {code}
> (4) However on removing the space before the set command - it works fine.
>
> *Analysis:*
> Looks like an issue with
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>
> In the runInternal() method, the untrimmed sql statement is split by space
> (\\s) but the substring operation to get the command args is done on the
> trimmed sql statement. This causes the issue.
> {code:java}
> String command = getStatement().trim();
> String[] tokens = statement.split("\\s");
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20608) Incorrect handling of sql command args in hive service leading to misleading error messages
[
https://issues.apache.org/jira/browse/HIVE-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637309#comment-16637309
]
Soumabrata Chakraborty commented on HIVE-20608:
---
Hi [~daijy] - There isn't any pre-existing test case for
org.apache.hive.service.cli.operation.HiveCommandOperation to which I can add.
Also there seems to be no clean way of testing just the changes made without
writing a test set-up for the whole class which seems pretty involved.
Please advise
> Incorrect handling of sql command args in hive service leading to misleading
> error messages
> ---
>
> Key: HIVE-20608
> URL: https://issues.apache.org/jira/browse/HIVE-20608
> Project: Hive
> Issue Type: Bug
>Reporter: Soumabrata Chakraborty
>Assignee: Soumabrata Chakraborty
>Priority: Major
> Attachments: HIVE-20608.patch
>
>
> *Steps to reproduce:*
> (1) Connect to HiveServer2 using JDBC driver (not via Beeline)
> (2) Execute a set command with a space before set – e.g. " set
> hive.exec.dynamic.partiton=true"
> (3) The error that is returned says:
> {code:java}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: Cannot modify set hive.exec.dynamic.partition at
> runtime. It is not in list of params that are allowed to be modified at
> runtime
> {code}
> (4) However on removing the space before the set command - it works fine.
>
> *Analysis:*
> Looks like an issue with
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java]
>
> In the runInternal() method, the untrimmed sql statement is split by space
> (\\s) but the substring operation to get the command args is done on the
> trimmed sql statement. This causes the issue.
> {code:java}
> String command = getStatement().trim();
> String[] tokens = statement.split("\\s");
> String commandArgs = command.substring(tokens[0].length()).trim();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Attachment: HIVE-20660.3.patch
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Open (was: Patch Available)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Patch Available (was: Open)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637301#comment-16637301
]
Alexander Kolbasov commented on HIVE-20671:
---
Interesting - looks like now the streaming tests compile and run but are broken!
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637299#comment-16637299
]
Alexander Kolbasov commented on HIVE-20671:
---
[~jmarhuen] I was able to reproduce the original problem and don't see it with
the patch.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20676) HiveServer2: PrivilegeSynchronizer is not set to daemon status
[
https://issues.apache.org/jira/browse/HIVE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637277#comment-16637277
]
Hive QA commented on HIVE-20676:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14206/dev-support/hive-personality.sh
|
| git revision | master / a06a370 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14206/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2: PrivilegeSynchronizer is not set to daemon status
> --
>
> Key: HIVE-20676
> URL: https://issues.apache.org/jira/browse/HIVE-20676
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20676.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20590) Allow merge statement to have column schema
[ https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20590: -- Status: Open (was: Patch Available) > Allow merge statement to have column schema > --- > > Key: HIVE-20590 > URL: https://issues.apache.org/jira/browse/HIVE-20590 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Vineet Garg >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch > > > Currently MERGE statement doesn't let user specify column schema with INSERT > statements, therefore DEFAULT constraint are not applicable with it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20590) Allow merge statement to have column schema
[ https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20590: -- Status: Patch Available (was: Open) > Allow merge statement to have column schema > --- > > Key: HIVE-20590 > URL: https://issues.apache.org/jira/browse/HIVE-20590 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Vineet Garg >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch > > > Currently MERGE statement doesn't let user specify column schema with INSERT > statements, therefore DEFAULT constraint are not applicable with it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20590) Allow merge statement to have column schema
[ https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20590: -- Attachment: HIVE-20590.03.patch > Allow merge statement to have column schema > --- > > Key: HIVE-20590 > URL: https://issues.apache.org/jira/browse/HIVE-20590 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Vineet Garg >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch > > > Currently MERGE statement doesn't let user specify column schema with INSERT > statements, therefore DEFAULT constraint are not applicable with it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637254#comment-16637254
]
Matt McCline commented on HIVE-20563:
-
Things to look at:
1) If a THEN/ELSE expression is NULL the new data type conversion code inserts
a "ConstantVectorExpression(val null)" so the resulting expression will be
IfExprCondExprCondExpr instead of the optimized IfExprCondExprNull, for example.
2) A query now does not vectorize due to "notVectorizedReason: SELECT operator:
Unsupported type timestamp for cast to String" where it use to vectorize with
VectorUDFAdaptor. Suppress the exception and fall back to VectorUDFAdaptor.
Also, consider supporting that conversion.
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
>
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637243#comment-16637243
]
Hive QA commented on HIVE-20306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942220/HIVE-20306.17.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14205/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14205/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942220/HIVE-20306.17.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942220 - PreCommit-HIVE-Build
> Implement projection spec for fetching only requested fields from partitions
>
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch,
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch,
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch,
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch,
> HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch,
> HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637241#comment-16637241
]
Hive QA commented on HIVE-20306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942220/HIVE-20306.17.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15043 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14204/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14204/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942220 - PreCommit-HIVE-Build
> Implement projection spec for fetching only requested fields from partitions
>
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch,
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch,
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch,
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch,
> HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch,
> HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637171#comment-16637171
]
Hive QA commented on HIVE-20306:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
19s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 53 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} standalone-metastore/metastore-server generated 7 new
+ 179 unchanged - 3 fixed = 186 total (was 182) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to filterSpec in
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(GetPartitionsRequest)
At
HiveMetaStore.java:org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(GetPartitionsRequest)
At HiveMetaStore.java:[line 4681] |
| | Dead store to confLocation in new
org.apache.hadoop.hive.metastore.MetaStoreDirectSql(PersistenceManager,
Configuration, String) At MetaStoreDirectSql.java:new
org.apache.hadoop.hive.metastore.MetaStoreDirectSql(PersistenceManager,
Configuration, String) At MetaStoreDirectSql.java:[line 207] |
| |
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlBoolean(Object)
has Boolean return type and returns explicit null At
MetastoreDirectSqlUtils.java:and returns explicit null At
MetastoreDirectSqlUtils.java:[line 514] |
| |
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(Query,
Object[], String) concatenates strings using + in a loop At
MetastoreDirectSqlUtils.java:using + in a loop At
MetastoreDirectSqlUtils.java:[line 67] |
| | org.apache.hadoop.hive.metastore.ObjectStore$16.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8727] |
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.setSingleValuedFields(List,
TreeMap, TreeMap, TreeMap, TreeMap) At
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637159#comment-16637159 ] Zoltan Haindrich commented on HIVE-20638: - +1 pending tests - we are taking green test results seriously ... I've rescheduled a ptest run :) > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch, HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20638: --- Assignee: Zoltan Haindrich (was: Laszlo Bodor) > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch, HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20638: Attachment: HIVE-20638.03.patch > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch, HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-20638: --- Assignee: Laszlo Bodor (was: Zoltan Haindrich) > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch, HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP reconnect to already running Yarn app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Description:
Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
llap yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't (re)connecting and making use of the running llap app.
It's also possible that the containers were simply slow to allocate as the
cluster was operating at 100% capacity and therefore weren't fully initialized
when the app failed, but the error feedback doesn't give enough details about
the state of the llap0 app.
was:
Improve HiveServer2 Interactive LLAP to reconnect to already running hive llap
yarn app.
Currently HiveServer2 Interactive startup may fail with the following error if
it cannot get enough containers on the queue:
{code:java}
WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
state RUNNING is attained.
2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
Desired Instance : '4'
2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
2018-10-01 16:26:55,625 - Stopping LLAP
2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
True, 'user': 'hive', 'stderr': -1}{code}
I could meanwhile see 5 containers for a previous hive llap invocation in the
yarn scheduler page and this is the only HiveServer2 Interactive instance, so
it appears it wasn't reconnecting and making use of the running llap app.
> HiveServer2 Interactive LLAP reconnect to already running Yarn app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't (re)connecting and making use of the running llap app.
> It's also possible that the containers were simply slow to allocate as the
> cluster was operating at 100% capacity and therefore weren't fully
> initialized when the app failed, but the error feedback doesn't give enough
> details about the state of the llap0 app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20666) HiveServer2 Interactive LLAP (re)connect to already running Yarn llap0 app
[
https://issues.apache.org/jira/browse/HIVE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HIVE-20666:
---
Summary: HiveServer2 Interactive LLAP (re)connect to already running Yarn
llap0 app (was: HiveServer2 Interactive LLAP reconnect to already running Yarn
app)
> HiveServer2 Interactive LLAP (re)connect to already running Yarn llap0 app
> --
>
> Key: HIVE-20666
> URL: https://issues.apache.org/jira/browse/HIVE-20666
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
>Affects Versions: 1.2.1
>Reporter: Hari Sekhon
>Priority: Major
>
> Improve HiveServer2 Interactive LLAP to (re)connect to already running hive
> llap yarn app.
> Currently HiveServer2 Interactive startup may fail with the following error
> if it cannot get enough containers on the queue:
> {code:java}
> WARN cli.LlapStatusServiceDriver: Watch timeout 200s exhausted before desired
> state RUNNING is attained.
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' in 'RUNNING_PARTIAL' state. Live
> Instances : '3'. Desired Instances : '4' after 212.498996019 secs.
> 2018-10-01 16:26:55,624 - App state is RUNNING_PARTIAL. Live Instances : '3',
> Desired Instance : '4'
> 2018-10-01 16:26:55,624 - LLAP app 'llap0' deployment unsuccessful.
> 2018-10-01 16:26:55,625 - Stopping LLAP
> 2018-10-01 16:26:55,625 - call[['slider', 'stop', u'llap0']] {'logoutput':
> True, 'user': 'hive', 'stderr': -1}{code}
> I could meanwhile see 5 containers for a previous hive llap invocation in the
> yarn scheduler page and this is the only HiveServer2 Interactive instance, so
> it appears it wasn't (re)connecting and making use of the running llap app.
> It's also possible that the containers were simply slow to allocate as the
> cluster was operating at 100% capacity and therefore weren't fully
> initialized when the app failed, but the error feedback doesn't give enough
> details about the state of the llap0 app.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20674) TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
[
https://issues.apache.org/jira/browse/HIVE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637122#comment-16637122
]
Hive QA commented on HIVE-20674:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942207/HIVE-20674.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15010 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14203/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14203/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14203/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942207 - PreCommit-HIVE-Build
> TestJdbcWithMiniLlapArrow.testKillQuery fail frequently
> ---
>
> Key: HIVE-20674
> URL: https://issues.apache.org/jira/browse/HIVE-20674
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20674.1.patch
>
>
> Intermittent but frequently:
> {code}
> java.sql.SQLException: org.apache.thrift.transport.TTransportException
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:227)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:243)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery(TestJdbcWithMiniLlapArrow.java:349)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
> at
>
[jira] [Updated] (HIVE-20669) JdbcStorageHandler push union of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20669:
---
Status: Patch Available (was: Open)
> JdbcStorageHandler push union of two different datasource to jdbc driver
>
>
> Key: HIVE-20669
> URL: https://issues.apache.org/jira/browse/HIVE-20669
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 UNION ALL SELECT * FROM ext_auth2
> {code}
> Calcite complain about unknown operation:
> {code}
> java.lang.AssertionError: Relational expression HepRelVertex#387 has
> calling-convention JDBC.DERBY but does not implement the required interface
> 'interface org.apache.calcite.adapter.jdbc.JdbcRel' of that convention
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:544)
> at org.apache.calcite.plan.RelOptRule.convert(RelOptRule.java:572)
> at
> org.apache.calcite.plan.RelOptRule.lambda$convertList$2(RelOptRule.java:607)
> at
> com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at
> com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
> at org.apache.calcite.rel.core.SetOp.(SetOp.java:61)
> at org.apache.calcite.rel.core.Union.(Union.java:43)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnion.(JdbcRules.java:708)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcUnionRule.convert(JdbcRules.java:697)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCUnionPushDownRule.onMatch(JDBCUnionPushDownRule.java:80)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:315)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2348)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1917)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1670)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1043)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1429)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:476)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:356)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:669)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1872)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1819)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1814)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at
[jira] [Updated] (HIVE-20677) JDBC storage handler ordering problem - single split flag
[ https://issues.apache.org/jira/browse/HIVE-20677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20677: --- Status: Patch Available (was: Open) > JDBC storage handler ordering problem - single split flag > - > > Key: HIVE-20677 > URL: https://issues.apache.org/jira/browse/HIVE-20677 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Gunther Hagleitner >Assignee: Jesus Camacho Rodriguez >Priority: Major > > When Calcite is pushing queries into the JDBC handler splitting the query via > offset/limit can cause issues (RDBMs is not guaranteed to return the data in > the same order every time.) > For these cases we want to: > a) Add a "do not split" flag to the jdbc handler. In that mode jdbc handler > will skip the count and offset/limit processing but just run the query in a > single node. Flag default will be false. > b) Have calcite automatically set this flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637114#comment-16637114
]
Bharathkrishna Guruvayoor Murali commented on HIVE-20610:
-
Oops, checkstyle errors. Fixed them in patch 4.
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
[jira] [Updated] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20610:
Attachment: HIVE-20610.4.patch
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.drop_table_with_environment_context(Unknown
> Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:3204)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1492)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:1432)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable(TestDbNotificationListener.java:522)
> [test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]{code}
>
>
--
This
[jira] [Assigned] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-20686: --- > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20685) XmlOutputFormat does not escape all required symbols in case of attribte/text
[ https://issues.apache.org/jira/browse/HIVE-20685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20685 started by Dmitrii. -- > XmlOutputFormat does not escape all required symbols in case of attribte/text > - > > Key: HIVE-20685 > URL: https://issues.apache.org/jira/browse/HIVE-20685 > Project: Hive > Issue Type: Bug >Reporter: Dmitrii >Assignee: Dmitrii >Priority: Minor > > Currently it escapes only " and < while & also must be escaped in case of > attribute/text based on [1]. > At the same time both " and ' are not required to be encoded for case of > element text while for xml attribute one of them should be depending on > attribute quote type. > Also > must be escaped if there is a sequence ]]> inside text or attribute > value which is not a part of CDATA > [1] https://www.w3.org/TR/2008/REC-xml-20081126/#syntax -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20685) XmlOutputFormat does not escape all required symbols in case of attribte/text
[ https://issues.apache.org/jira/browse/HIVE-20685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitrii reassigned HIVE-20685: -- > XmlOutputFormat does not escape all required symbols in case of attribte/text > - > > Key: HIVE-20685 > URL: https://issues.apache.org/jira/browse/HIVE-20685 > Project: Hive > Issue Type: Bug >Reporter: Dmitrii >Assignee: Dmitrii >Priority: Minor > > Currently it escapes only " and < while & also must be escaped in case of > attribute/text based on [1]. > At the same time both " and ' are not required to be encoded for case of > element text while for xml attribute one of them should be depending on > attribute quote type. > Also > must be escaped if there is a sequence ]]> inside text or attribute > value which is not a part of CDATA > [1] https://www.w3.org/TR/2008/REC-xml-20081126/#syntax -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20681) Support custom path filter for ORC tables
[ https://issues.apache.org/jira/browse/HIVE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637099#comment-16637099 ] Eugene Koifman commented on HIVE-20681: --- Could you give a concrete example of some files on disk and what filter you'd like to generate? > Support custom path filter for ORC tables > - > > Key: HIVE-20681 > URL: https://issues.apache.org/jira/browse/HIVE-20681 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Igor Kryvenko >Assignee: Igor Kryvenko >Priority: Minor > > Currently, Orc file input format does not take in path filters set in the > property "mapreduce.input.pathfilter.class" OR " > mapred.input.pathfilter.class ". So, we cannot use custom filters with Orc > files. > AcidUtils class has a static filter called "hiddenFilters" which is used by > ORC to filter input paths. If we can pass the custom filter classes(set in > the property mentioned above) to AcidUtils and replace hiddenFilter with a > filter that does an "and" operation over hiddenFilter+customFilters, the > filters would work well. > It would be useful to have the ability to filter out rows based on > path/filenames, current ORC features like bloom filters and indexes are not > good enough for them to minimize the number of disk read operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637094#comment-16637094 ] Jaume M commented on HIVE-20672: Can you review [~sershe] > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20672.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20670) ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of local variable
[ https://issues.apache.org/jira/browse/HIVE-20670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20670: --- Attachment: HIVE-20670.2.patch Status: Patch Available (was: Open) > ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of > local variable > -- > > Key: HIVE-20670 > URL: https://issues.apache.org/jira/browse/HIVE-20670 > Project: Hive > Issue Type: Bug > Components: HiveServer2, llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20670.1.patch, HIVE-20670.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
