[jira] [Commented] (HIVE-22485) Cross product should set the conf in UnorderedPartitionedKVEdgeConfig
[
https://issues.apache.org/jira/browse/HIVE-22485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973094#comment-16973094
]
Hive QA commented on HIVE-22485:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19397/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19397/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cross product should set the conf in UnorderedPartitionedKVEdgeConfig
> -
>
> Key: HIVE-22485
> URL: https://issues.apache.org/jira/browse/HIVE-22485
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-22485.1.patch
>
>
> SSL and other options would not be sent correctly, if this is not setup.
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L545
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22486 started by Jesus Camacho Rodriguez. -- > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22486.patch > > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22486: --- Status: Patch Available (was: In Progress) > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22486.patch > > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22486: --- Attachment: HIVE-22486.patch > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22486.patch > > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22486: -- > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973077#comment-16973077
]
Hive QA commented on HIVE-22461:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985666/HIVE-22461.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19396/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19396/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985666 - PreCommit-HIVE-Build
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
>
[jira] [Updated] (HIVE-22477) Avro logical type timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-22477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjar Akhmedov updated HIVE-22477:
---
Attachment: HIVE-22477.3.patch
> Avro logical type timestamp conversion is slow
> --
>
> Key: HIVE-22477
> URL: https://issues.apache.org/jira/browse/HIVE-22477
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
> Environment: Hive 3.1.0
>Reporter: Sanjar Akhmedov
>Priority: Major
> Labels: Performance
> Attachments: HIVE-22477.1.patch, HIVE-22477.2.patch,
> HIVE-22477.3.patch, flamegraph.svg
>
>
> We have an avro backed table with hundreds of billions timestamps. Simple
> {{SELECT COUNT(*) FROM t}} query takes many hours to complete in version
> 3.1.0 versus tens of minutes in version 1.2.1.
> Looking at the attached flamegraph of one of the yarn containers, hive is
> spending most of the time throwing exceptions during avro timestamp
> conversion.
> It is generally good idea to avoid throwing exceptions in performance
> critical sections, as exception creation is an expensive operation, and
> potentially repeating for many rows/values in a query can have drastic
> performance implications.
> Afaics there is no reason to convert numeric timestamp to a string and enter
> very lenient
> {{org.apache.hadoop.hive.common.type.TimestampTZUtil#parse(java.lang.String,
> java.time.ZoneId)}} to do timezone conversion.
> This patch changes the conversion of {{Date}} and {{Timestamp}} to
> {{TimestampTZ}} such that it doesn't invoke {{parse}}.
> JMH timings before:
> {code:java}
> Benchmark Mode Cnt Score Error
> Units
> TimestampTZUtilBench.convertDate avgt2 10091.990
> ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 10657.596
> ns/op
> {code}
> JMH timings after:
> {code:java}
> Benchmark Mode Cnt Score Error Units
> TimestampTZUtilBench.convertDate avgt2 48.371 ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 51.170 ns/op
> {code}
> JMH stack profile before:
> {code:java}
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertDate:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 97.4% 97.4% java.lang.Throwable.fillInStackTrace
> 1.6% 1.6% java.time.format.DateTimeFormatter.parse
> 0.2% 0.2% java.time.ZoneId.from
> 0.1% 0.1% java.util.HashMap.hash
> 0.1% 0.1% java.lang.Number.
> 0.1% 0.1%
> java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format
> 0.1% 0.1% java.lang.StringBuilder.append
> 0.1% 0.1% java.util.HashMap.putVal
> 0.1% 0.1% java.lang.String.valueOf
> 0.1% 0.1% java.util.regex.Pattern$BmpCharProperty.match
> 0.2% 0.2%
> ...
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertTimestamp:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 96.5% 96.5% java.lang.Throwable.fillInStackTrace
> 1.0% 1.0% java.time.format.DateTimeFormatter.parse
> 0.6% 0.6% org.apache.hadoop.hive.common.type.TimestampTZUtil.parse
> 0.4% 0.4% java.time.ZoneId.from
> 0.2% 0.2%
> java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format
> 0.2% 0.2% java.time.format.Parsed.resolveFields
> 0.2% 0.2% java.lang.String.valueOf
> 0.1% 0.1% java.lang.StringBuilder.append
> 0.1% 0.1% java.util.HashMap.hash
> 0.1% 0.1% java.time.format.DateTimeParseContext.toResolved
> 0.6% 0.6%
> {code}
> JMH stack profile after:
> {code:java}
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertDate:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 91.6% 91.6% java.time.ZonedDateTime.ofInstant
> 8.0% 8.0%
> org.apache.hive.benchmark.common.generated.TimestampTZUtilBench_convertDate_jmhTest.convertDate_avgt_jmhStub
> 0.1% 0.1% java.time.zone.ZoneRules.
> 0.1% 0.1% java.time.LocalDateTime.ofEpochSecond
> 0.1% 0.1% org.apache.hadoop.hive.common.type.TimestampTZUtil.convert
> 0.1% 0.1% java.time.LocalDate.ofEpochDay
>
[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[
https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973065#comment-16973065
]
Rui Li commented on HIVE-22158:
---
Thanks [~ngangam]. Setting {{metastore.warehouse.external.dir}} solves my issue.
> HMS Translation layer - Disallow non-ACID MANAGED tables.
> -
>
> Key: HIVE-22158
> URL: https://issues.apache.org/jira/browse/HIVE-22158
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22158.1.patch, HIVE-22158.1.patch,
> HIVE-22158.2.patch
>
>
> In the recent commits, we have allowed non-ACID MANAGED tables to be created
> by clients that have some form of ACID WRITE capabilities.
> I think it would make sense to disallow this entirely. MANAGED tables should
> be ACID tables only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22081) Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there are too many Table/partitions are eligible for compaction
[ https://issues.apache.org/jira/browse/HIVE-22081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973063#comment-16973063 ] Vipin Vishvkarma commented on HIVE-22081: - [~Rajkumar Singh] Will, there be any performance improvement with this change, as I don't see changes related to point 2 from the description in the final change and we have used stream() which is sequential in nature. I may be missing something here, can you please confirm. > Hivemetastore Performance: Compaction Initiator Thread overwhelmed if there > are too many Table/partitions are eligible for compaction > -- > > Key: HIVE-22081 > URL: https://issues.apache.org/jira/browse/HIVE-22081 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21917.01.patch, HIVE-21917.02.patch, > HIVE-21917.03.patch, HIVE-22081.04.patch, HIVE-22081.patch > > > if Automatic Compaction is turned on, Initiator thread check for potential > table/partitions which are eligible for compactions and run some checks in > for loop before requesting compaction for eligibles. Though initiator thread > is configured to run at interval 5 min default, in case of many objects it > keeps on running as these checks are IO intensive and hog cpu. > In the proposed changes, I am planning to do > 1. passing less object to for loop by filtering out the objects based on the > condition which we are checking within the loop. > 2. Doing Async call using future to determine compaction type(this is where > we do FileSystem calls) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973043#comment-16973043
]
Hive QA commented on HIVE-22461:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 374 unchanged - 34 fixed = 375 total (was 408) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 177 unchanged - 1 fixed = 177 total (was 178) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19396/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19396/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19396/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973032#comment-16973032
]
Hive QA commented on HIVE-22476:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985664/HIVE-22476.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17708 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_udf]
(batchId=128)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFLastDay.testWrongTsStr
(batchId=316)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFNextDay.testNotValidValues
(batchId=318)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19395/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19395/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19395/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985664 - PreCommit-HIVE-Build
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973017#comment-16973017
]
Hive QA commented on HIVE-22476:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19395/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19395/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from
[jira] [Commented] (HIVE-19805) TableScanDesc Use Commons Library
[
https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973006#comment-16973006
]
Hive QA commented on HIVE-19805:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985658/HIVE-19805.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19394/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19394/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19394/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:25:10.252
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19394/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:25:10.255
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:25:10.982
+ rm -rf ../yetus_PreCommit-HIVE-Build-19394
+ mkdir ../yetus_PreCommit-HIVE-Build-19394
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19394
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19394/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java: does
not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc3248928863472048616.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc3248928863472048616.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-pre-upgrade: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-pre-upgrade
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19394
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985658 - PreCommit-HIVE-Build
> TableScanDesc Use Commons Library
>
[jira] [Commented] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973005#comment-16973005
]
Hive QA commented on HIVE-18882:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12950181/HIVE-18882.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19393/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19393/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19393/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:23:49.814
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19393/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:23:49.817
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-13 04:23:50.978
+ rm -rf ../yetus_PreCommit-HIVE-Build-19393
+ mkdir ../yetus_PreCommit-HIVE-Build-19393
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19393
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19393/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
does not exist in index
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:533
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:533
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
with conflicts.
U
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19393
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12950181 - PreCommit-HIVE-Build
> Do Not Hide Exception in Hive Metastore Client Connection
> -
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: Manoj Narayanan
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if
[jira] [Commented] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973003#comment-16973003
]
Hive QA commented on HIVE-16879:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985656/HIVE-16879.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17701 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
(batchId=208)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
org.apache.hive.service.TestDFSErrorHandling.testAccessDenied (batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19392/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19392/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19392/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985656 - PreCommit-HIVE-Build
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22485) Cross product should set the conf in UnorderedPartitionedKVEdgeConfig
[ https://issues.apache.org/jira/browse/HIVE-22485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22485: Status: Patch Available (was: Open) > Cross product should set the conf in UnorderedPartitionedKVEdgeConfig > - > > Key: HIVE-22485 > URL: https://issues.apache.org/jira/browse/HIVE-22485 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-22485.1.patch > > > SSL and other options would not be sent correctly, if this is not setup. > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L545 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22485) Cross product should set the conf in UnorderedPartitionedKVEdgeConfig
[ https://issues.apache.org/jira/browse/HIVE-22485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-22485: --- Assignee: Rajesh Balamohan > Cross product should set the conf in UnorderedPartitionedKVEdgeConfig > - > > Key: HIVE-22485 > URL: https://issues.apache.org/jira/browse/HIVE-22485 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-22485.1.patch > > > SSL and other options would not be sent correctly, if this is not setup. > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L545 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972978#comment-16972978
]
Hive QA commented on HIVE-16879:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 15 unchanged - 3 fixed = 15 total (was 18) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19392/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19392/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972966#comment-16972966
]
Hive QA commented on HIVE-22484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985654/HIVE-22484.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19391/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19391/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19391/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985654 - PreCommit-HIVE-Build
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972950#comment-16972950
]
Hive QA commented on HIVE-22484:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
11s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 16 unchanged -
1 fixed = 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19391/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql service jdbc U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19391/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
[jira] [Commented] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972931#comment-16972931
]
Hive QA commented on HIVE-22408:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985650/HIVE-22408.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17708 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=248)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=276)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=276)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testCustomPathsForCTLV (batchId=288)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19390/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19390/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19390/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985650 - PreCommit-HIVE-Build
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch
>
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-22461:
Attachment: HIVE-22461.5.patch
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch, HIVE-22461.5.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
>
[jira] [Updated] (HIVE-22485) Cross product should set the conf in UnorderedPartitionedKVEdgeConfig
[ https://issues.apache.org/jira/browse/HIVE-22485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22485: Attachment: HIVE-22485.1.patch > Cross product should set the conf in UnorderedPartitionedKVEdgeConfig > - > > Key: HIVE-22485 > URL: https://issues.apache.org/jira/browse/HIVE-22485 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-22485.1.patch > > > SSL and other options would not be sent correctly, if this is not setup. > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L545 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972910#comment-16972910
]
Hive QA commented on HIVE-22408:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 2 unchanged - 22
fixed = 2 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19390/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19390/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch
>
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22476) Hive datediff function provided inconsistent results when hive.fetch.task.conversion is set to none
[
https://issues.apache.org/jira/browse/HIVE-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22476:
--
Attachment: HIVE-22476.2.patch
> Hive datediff function provided inconsistent results when
> hive.fetch.task.conversion is set to none
> ---
>
> Key: HIVE-22476
> URL: https://issues.apache.org/jira/browse/HIVE-22476
> Project: Hive
> Issue Type: Bug
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22476.2.patch, HIVE-22476.patch
>
>
> The actual issue stems to the different date parser used by various part of
> the engine.
> Fetch task uses udfdatediff via {code}
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate{code} while the
> vectorized llap execution uses {code}VectorUDFDateDiffScalarCol{code}.
> This fix is meant to be not very intrusive and will add more support to the
> GenericUDFToDate by enhancing the parser.
> For the longer term will be better to use one parser for all the operators.
> Thanks [~Rajkumar Singh] for the repro example
> {code}
> create external table testdatediff(datetimecol string) stored as orc;
> insert into testdatediff values ('2019-09-09T10:45:49+02:00'),('2019-07-24');
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> set hive.ferch.task.conversion=none;
> select datetimecol from testdatediff where datediff(cast(current_timestamp as
> string), datetimecol)<183;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972905#comment-16972905
]
Hive QA commented on HIVE-22461:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985642/HIVE-22461.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19389/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19389/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19389/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:34:51.160
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19389/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:34:51.163
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:34:52.214
+ rm -rf ../yetus_PreCommit-HIVE-Build-19389
+ mkdir ../yetus_PreCommit-HIVE-Build-19389
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19389
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19389/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2782291993494432389.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2782291993494432389.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-0.23: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-0.23
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19389
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985642 -
[jira] [Commented] (HIVE-22479) Fix typo in GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972903#comment-16972903
]
Hive QA commented on HIVE-22479:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985640/HIVE-22479.001.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19388/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19388/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19388/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:33:30.006
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19388/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:33:30.010
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9678ca1 HIVE-22469: Lower Metastore DB Connection Pool Size in
QTests (David Mollitor, reviewed by Miklos Gergely)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-12 23:33:30.812
+ rm -rf ../yetus_PreCommit-HIVE-Build-19388
+ mkdir ../yetus_PreCommit-HIVE-Build-19388
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19388
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19388/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: corrupt patch at line 24
fatal: corrupt patch at line 24
fatal: corrupt patch at line 24
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19388
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985640 - PreCommit-HIVE-Build
> Fix typo in GenericUDF
> --
>
> Key: HIVE-22479
> URL: https://issues.apache.org/jira/browse/HIVE-22479
> Project: Hive
> Issue Type: Improvement
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: newbie, pull-request-available
> Attachments: HIVE-22479.001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> `Defered` corrected to `Deferred`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972901#comment-16972901
]
Hive QA commented on HIVE-22464:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985637/HIVE-22464.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19387/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19387/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19387/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985637 - PreCommit-HIVE-Build
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342259
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 22:55
Start Date: 12/Nov/19 22:55
Worklog Time Spent: 10m
Work Description: dlavati commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345489917
##
File path:
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java
##
@@ -336,7 +336,7 @@ protected void
authorizeDDLWork(HiveSemanticAnalyzerHookContext cntxt, Hive hive
} else if (ddlDesc instanceof AlterTableSetLocationDesc) {
AlterTableSetLocationDesc alterTable =
(AlterTableSetLocationDesc)ddlDesc;
Table table = hive.getTable(SessionState.get().getCurrentDatabase(),
- Utilities.getDbTableName(alterTable.getTableName())[1], false);
+ Utilities.getDbTableName(alterTable.getDbTableName())[1], false);
Review comment:
I've marked it deprecated this time.
Some places rely on some of the state changes from this, so killing it in 1
patch would have been harder.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342259)
Time Spent: 4h (was: 3h 50m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972857#comment-16972857
]
David Mollitor commented on HIVE-22484:
---
[~pvary] Where appropriate, I am passing them to SLF4J to allow it to handle
printing to the log. Otherwise, the exception is wrapped and it logged at a
higher level. Very few cases where {{printStackTrace}} makes sense.
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972854#comment-16972854
]
Hive QA commented on HIVE-22464:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} serde generated 0 new + 198 unchanged - 1 fixed =
198 total (was 199) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} ql generated 0 new + 1547 unchanged - 3 fixed = 1547
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19387/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19387/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972820#comment-16972820
]
Hive QA commented on HIVE-22453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985630/HIVE-22453.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19386/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19386/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19386/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985630 - PreCommit-HIVE-Build
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.2.patch, HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972802#comment-16972802
]
Zoltan Haindrich commented on HIVE-21198:
-
I think the patch already starts doing the right thing by changing the
signature of quite a few methods; as it's hard to cover this whole thing in a
single patch - I think we are ok to call this the first step and continue doing
it in follow-ups; I left some comments/questions/etc on the PR
+1
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972800#comment-16972800
]
Peter Vary commented on HIVE-22484:
---
[~belugabehr]: Could we have a pull request, or review board? I prefer to print
the stacktrace to logs for the cases where only the error message is returned.
Otherwise the source of the problem is lost. I think we should print out them
at least to the logs.
Thanks,
Peter
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22306) Use nonblocking thrift server for metastore
[ https://issues.apache.org/jira/browse/HIVE-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972799#comment-16972799 ] Qinghui Xu commented on HIVE-22306: --- Some update on the progress: I pushed a PR for THRIFT-4889, and we patched our hive metastore to use this PR. We are now testing this nonblocking hive metastore in our preprod. It's quite stable since several days, and our monitoring shows that it has smaller memory footprint and same cpu usage comparing to its blocking variant under a moderate load. We are going to run some load tests and to compare the behaviors. I would appreciate the help from the community to review my [thrift PR|[https://github.com/apache/thrift/pull/1892]] > Use nonblocking thrift server for metastore > --- > > Key: HIVE-22306 > URL: https://issues.apache.org/jira/browse/HIVE-22306 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Qinghui Xu >Priority: Major > > Currently hive metastore's threads are blocking for network io (it's using > `TThreadPoolServer` behind the scene), which means with increasing use cases > (in our tech stack there are different services relying on it, hiveserver2, > spark, presto, and more, all with a significant number of users), to handle > all connections it needs either a big thread pool or many instances with > smaller thread pools. And often, those metastores will see their thread pool > saturated, while the cpu usage is still quite low, just because most > connections stay idle and only run a query from time to time. This is thus a > great misuse of the computation resources. > Thus I propose to use a non blocking threading model, and run computation > asynchronously. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342173=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342173
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:26
Start Date: 12/Nov/19 21:26
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345401411
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
##
@@ -1127,139 +1127,142 @@ private void analyzeAlterTableLocation(ASTNode ast,
String tableName, Map partSpec)
+ private void analyzeAlterTablePartMergeFiles(ASTNode ast, TableName
tableName, Map partSpec)
throws SemanticException {
Path oldTblPartLoc = null;
Path newTblPartLoc = null;
Table tblObj = null;
ListBucketingCtx lbCtx = null;
-try {
Review comment:
earlier ; all exceptions (including any Runtime ones) were repackaged as
SemanticException; are we safe to remove this try?
update: I've just checked that we don't really experience any changes big
changes of this; Driver will catch it as a lasst resort
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342173)
Time Spent: 3h 50m (was: 3h 40m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342156
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345446033
##
File path:
ql/src/test/results/clientpositive/encrypted/encryption_move_tbl.q.out
##
@@ -109,8 +109,8 @@ POSTHOOK: query: SHOW TABLES
POSTHOOK: type: SHOWTABLES
POSTHOOK: Input: database:default
src
-FAILED: SemanticException [Error 10001]: Table not found encrypted_table_n1
-FAILED: SemanticException [Error 10001]: Table not found encrypted_table_n1
+FAILED: SemanticException [Error 10001]: Table not found
default.encrypted_table_n1
Review comment:
it's good to see that it's getting more consistend; but why do we see this 3
times? :D
I don't fully understand whats going on hereFAILED during a
clientpositive test...why is that okay? :dagger:
(this is clearly outside of the scope of this change)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342156)
Time Spent: 3.5h (was: 3h 20m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342166
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345399636
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
##
@@ -897,10 +897,10 @@ private void analyzeAlterTableProps(String[] qualified,
HashMap
}
// if table is being modified to be external we need to make sure
existing table
// doesn't have enabled constraint since constraints are disallowed with
such tables
- else if(entry.getKey().equals("external") &&
entry.getValue().equals("true")){
-if(hasConstraintsEnabled(qualified[1])){
+ else if (entry.getKey().equals("external") &&
entry.getValue().equals("true")) {
+if (hasConstraintsEnabled(tableName.getTable())) {
Review comment:
this seems fishy to me...I guess if we hit this line when we are issuing an
alter on a table of different database; things might go south...or is there
something I'm missing?
(this is of course outside of the scope of this ticket)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342166)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342159
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345393971
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
##
@@ -1160,16 +1158,16 @@ protected static void processForeignKeys(String
databaseName, String tableName,
" The number of foreign key columns should be same as number of parent
key columns "));
}
-String[] parentDBTbl = getQualifiedTableName((ASTNode)
child.getChild(ptIndex));
+final TableName parentTblName = getQualifiedTableName((ASTNode)
child.getChild(ptIndex));
for (int j = 0; j < child.getChild(fkIndex).getChildCount(); j++) {
SQLForeignKey sqlForeignKey = new SQLForeignKey();
- sqlForeignKey.setFktable_db(databaseName);
- sqlForeignKey.setFktable_name(tableName);
+ sqlForeignKey.setFktable_db(tName.getDb());
+ sqlForeignKey.setFktable_name(tName.getTable());
Tree fkgrandChild = child.getChild(fkIndex).getChild(j);
checkColumnName(fkgrandChild.getText());
sqlForeignKey.setFkcolumn_name(unescapeIdentifier(fkgrandChild.getText().toLowerCase()));
- sqlForeignKey.setPktable_db(parentDBTbl[0]);
- sqlForeignKey.setPktable_name(parentDBTbl[1]);
+ sqlForeignKey.setPktable_db(parentTblName.getDb());
Review comment:
at some point we might want to consider changing the thrift as well?
or we should leave that alone?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342159)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342158
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345405314
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -2296,16 +2297,16 @@ private void getMetaData(QB qb, ReadEntity parentInput)
}
if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVESTATSAUTOGATHER)) {
// Add the table spec for the destination table.
- qb.getParseInfo().addTableSpec(ts.tableName.toLowerCase(), ts);
+
qb.getParseInfo().addTableSpec(ts.getTableName().getTable().toLowerCase(), ts);
Review comment:
I think this means that we are using only the table's name here; without
database/etc qualificationthe patch right now preserves the existing
behaviour; but to be more correct: we should be using either TableName's fqdn
toString - please open a followup; I suspect that this might cause incorrect
behaviour as well; I also think that addTableSpec should accept TableName; and
do this magic there instead of at every call site; I just seen another
`[...].toLowerCase()` line repeated a few lines below
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342158)
Time Spent: 3.5h (was: 3h 20m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342157
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345403274
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -2275,9 +2276,9 @@ private void getMetaData(QB qb, ReadEntity parentInput)
// Whether we are using an acid compliant transaction manager has
already been caught in
// UpdateDeleteSemanticAnalyzer, so if we are updating or deleting
and getting nonAcid
// here, it means the table itself doesn't support it.
-throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE,
ts.tableName);
+throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE,
ts.getTableName().getTable());
Review comment:
we might want to consider reporting the fully qualified table references in
exceptions; so that they are more precise (cn be done probably later)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342157)
Time Spent: 3.5h (was: 3h 20m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342154
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345391449
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
##
@@ -394,48 +394,53 @@ public static String getUnescapedName(ASTNode
tableOrColumnNode, String currentD
if (tokenType == HiveParser.TOK_TABNAME) {
// table node
Map.Entry dbTablePair =
getDbTableNamePair(tableOrColumnNode);
- String dbName = dbTablePair.getKey();
- String tableName = dbTablePair.getValue();
- if (dbName != null){
-return dbName + "." + tableName;
Review comment:
it's good to see this going away! :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342154)
Time Spent: 3h 20m (was: 3h 10m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342155
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345441151
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/AddPrimaryKeyHandler.java
##
@@ -53,16 +54,17 @@
return tasks;
}
-String actualDbName = context.isDbNameEmpty() ? pks.get(0).getTable_db() :
context.dbName;
-String actualTblName = pks.get(0).getTable_name();
+final String actualDbName = context.isDbNameEmpty() ?
pks.get(0).getTable_db() : context.dbName;
+final String actualTblName = pks.get(0).getTable_name();
+final TableName tName = TableName.fromString(actualTblName, null,
actualDbName);
Review comment:
can we chane the order of the arguments of this method?
cat, db, tblName instead (can be done in a followup)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342155)
Time Spent: 3.5h (was: 3h 20m)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342161
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345401411
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
##
@@ -1127,139 +1127,142 @@ private void analyzeAlterTableLocation(ASTNode ast,
String tableName, Map partSpec)
+ private void analyzeAlterTablePartMergeFiles(ASTNode ast, TableName
tableName, Map partSpec)
throws SemanticException {
Path oldTblPartLoc = null;
Path newTblPartLoc = null;
Table tblObj = null;
ListBucketingCtx lbCtx = null;
-try {
Review comment:
earlier ; all exceptions (including any Runtime ones) were repackaged as
SemanticException; are we safe to remove this try?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342161)
Time Spent: 3h 40m (was: 3.5h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342160
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345446635
##
File path: ql/src/test/results/clientpositive/input10.q.out
##
@@ -21,7 +21,7 @@ STAGE DEPENDENCIES:
STAGE PLANS:
Stage: Stage-0
Describe Table
- table: TEST10
+ table: default.TEST10
Review comment:
:+1:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342160)
Time Spent: 3h 40m (was: 3.5h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342162
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345441875
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/RenameTableHandler.java
##
@@ -55,18 +55,19 @@
}
}
- String oldName = StatsUtils.getFullyQualifiedTableName(oldDbName,
tableObjBefore.getTableName());
- String newName = StatsUtils.getFullyQualifiedTableName(newDbName,
tableObjAfter.getTableName());
+ TableName oldName = TableName.fromString(tableObjBefore.getTableName(),
null, oldDbName);
+ TableName newName = TableName.fromString(tableObjAfter.getTableName(),
null, newDbName);
ReplicationSpec replicationSpec = context.eventOnlyReplicationSpec();
if (ReplUtils.isTableMigratingToTransactional(context.hiveConf,
tableObjAfter)) {
replicationSpec.setMigratingToTxnTable();
}
- AlterTableRenameDesc renameTableDesc = new AlterTableRenameDesc(oldName,
replicationSpec, false, newName);
+ AlterTableRenameDesc renameTableDesc =
+ new AlterTableRenameDesc(oldName, replicationSpec, false,
newName.getNotEmptyDbTable());
Review comment:
the newName is passed as string; while oldName is a TableName (ok in a
followup as well)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342162)
Time Spent: 3h 40m (was: 3.5h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342163
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345443647
##
File path: ql/src/test/results/clientnegative/delete_non_acid_table.q.out
##
@@ -34,4 +34,4 @@ POSTHOOK: Input: default@not_an_acid_table2
-10708830710ruyd6Y50JpdGRf6HqD
-1070551679iUR3Q
-1069736047k17Am8uPHWk02cEf1jet
-FAILED: SemanticException [Error 10297]: Attempt to do update or delete on
table default.not_an_acid_table2 that is not transactional
+FAILED: SemanticException [Error 10297]: Attempt to do update or delete on
table not_an_acid_table2 that is not transactional
Review comment:
we are loosing the qualified database name from the exception; I think this
should be made consistent; to show it everywhere - I'm not sure about if we
should be also showing the catalog name or not... but if we do; we should "try"
to be consistent about that as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342163)
Time Spent: 3h 40m (was: 3.5h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342165
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345449867
##
File path: storage-api/src/java/org/apache/hadoop/hive/common/TableName.java
##
@@ -86,36 +101,55 @@ public String getTable() {
/**
* Get the name in db.table format, for use with stuff not yet converted to
use the catalog.
+ * Fair warning, that if the db is null, this will return null.tableName
+ * @deprecated use {@link #getNotEmptyDbTable()} instead.
*/
+ // to be @Deprecated
public String getDbTable() {
return db + DatabaseName.CAT_DB_TABLE_SEPARATOR + table;
+ }
+
+ /**
+ * Get the name in `db`.`table` escaped format, if db is not empty,
otherwise pass only the table name.
+ */
+ public String getEscapedNotEmptyDbTable() {
Review comment:
I think these escaper things should live in the exec side; aren't they only
used during unparsing around there?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342165)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342164
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:21
Start Date: 12/Nov/19 21:21
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345448803
##
File path: storage-api/src/java/org/apache/hadoop/hive/common/TableName.java
##
@@ -17,34 +17,45 @@
*/
package org.apache.hadoop.hive.common;
+import java.io.Serializable;
+import java.util.Objects;
+
/**
* A container for a fully qualified table name, i.e.
catalogname.databasename.tablename. Also
* includes utilities for string parsing.
*/
-public class TableName {
+public class TableName implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ /** Exception message thrown. */
+ private static final String ILL_ARG_EXCEPTION_MSG =
Review comment:
I don't really see the benefit of making this a static final string;
but anyway - I think the end goal is to stop the usage of any `tableName`
which contains `dbname` or any other stuff...and in that case the method in
which this is used should go away at some point - am I right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342164)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?focusedWorklogId=342153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-342153
]
ASF GitHub Bot logged work on HIVE-21198:
-
Author: ASF GitHub Bot
Created on: 12/Nov/19 21:20
Start Date: 12/Nov/19 21:20
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #550: HIVE-21198
Introduce a database object reference class
URL: https://github.com/apache/hive/pull/550#discussion_r345389350
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1085,7 +1086,7 @@ public void createTable(Table tbl, boolean ifNotExists,
List checkConstraints)
throws HiveException {
try {
- if (tbl.getDbName() == null || "".equals(tbl.getDbName().trim())) {
+ if (org.apache.commons.lang3.StringUtils.isBlank(tbl.getDbName())) {
Review comment:
oh my we are using lang3.StringUtils, lang.StringUtils;
hadoop.common.StringUtils and hadoop.util.StringUtils
might be worth trying to use less ...
(can be fixed in next patch...this is not cruical)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 342153)
Time Spent: 3h 10m (was: 3h)
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21198.1.patch, HIVE-21198.10.patch,
> HIVE-21198.11.patch, HIVE-21198.2.patch, HIVE-21198.3.patch,
> HIVE-21198.4.patch, HIVE-21198.5.patch, HIVE-21198.6.patch,
> HIVE-21198.7.patch, HIVE-21198.8.patch, HIVE-21198.9.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972784#comment-16972784 ] David Mollitor commented on HIVE-22444: --- [~mgergely] Great! Thanks! I'll let this one set a few days before moving forward. > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.10.patch, > HIVE-22444.11.patch, HIVE-22444.2.patch, HIVE-22444.3.patch, > HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch, > HIVE-22444.7.patch, HIVE-22444.8.patch, HIVE-22444.9.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972771#comment-16972771
]
Hive QA commented on HIVE-22453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19386/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19386/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.2.patch, HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972762#comment-16972762 ] Miklos Gergely commented on HIVE-22444: --- ok, +1 > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.10.patch, > HIVE-22444.11.patch, HIVE-22444.2.patch, HIVE-22444.3.patch, > HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch, > HIVE-22444.7.patch, HIVE-22444.8.patch, HIVE-22444.9.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22484:
--
Description:
In many cases, the call to {{printStackTrace}} bypasses the logging framework,
in other cases, the error stack trace is printed and the exception is re-thrown
(log-and-throw is a bad pattern), and then there are some other edge cases.
Remove this call and replace with calls to the logging framework or remove
altogether if exception is wrapped and re-thrown.
was:
In many cases, the call to {{printStackTrace}} bypasses the logging framework,
in other cases, the error stack trace is printed and the exception is re-thrown
(log-and-throw is a bad pattern), and then there are some other edge cases.
Remove this call and replace with calls to the logging framework.
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-19805: -- Status: Open (was: Patch Available) > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch, > HIVE-19805.3.patch, Screenshot from 2018-06-09 11-37-08.png > > > Use commons library and remove some code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-19805: -- Status: Patch Available (was: Open) > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch, > HIVE-19805.3.patch, Screenshot from 2018-06-09 11-37-08.png > > > Use commons library and remove some code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-19805: -- Attachment: HIVE-19805.3.patch > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch, > HIVE-19805.3.patch, Screenshot from 2018-06-09 11-37-08.png > > > Use commons library and remove some code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-16879:
--
Attachment: (was: HIVE-16879.3.patch)
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972747#comment-16972747
]
David Mollitor commented on HIVE-18882:
---
[~abstractdog] Are you able to review this patch for me? Thanks!
> Do Not Hide Exception in Hive Metastore Client Connection
> -
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: Manoj Narayanan
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22423: -- Attachment: (was: HIVE-22423.1.patch) > Improve Logging In HadoopThriftAuthBridge > - > > Key: HIVE-22423 > URL: https://issues.apache.org/jira/browse/HIVE-22423 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22423.1.patch > > > # Remove superfluous debug log guards > # Improve messages > # Improve message format -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22477) Avro logical type timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-22477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972745#comment-16972745
]
Hive QA commented on HIVE-22477:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985629/HIVE-22477.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17708 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.schq.TestScheduledQueryIntegration.testScheduledQueryExecutionImpersonation
(batchId=279)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19385/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19385/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19385/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985629 - PreCommit-HIVE-Build
> Avro logical type timestamp conversion is slow
> --
>
> Key: HIVE-22477
> URL: https://issues.apache.org/jira/browse/HIVE-22477
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
> Environment: Hive 3.1.0
>Reporter: Sanjar Akhmedov
>Priority: Major
> Labels: Performance
> Attachments: HIVE-22477.1.patch, HIVE-22477.2.patch, flamegraph.svg
>
>
> We have an avro backed table with hundreds of billions timestamps. Simple
> {{SELECT COUNT(*) FROM t}} query takes many hours to complete in version
> 3.1.0 versus tens of minutes in version 1.2.1.
> Looking at the attached flamegraph of one of the yarn containers, hive is
> spending most of the time throwing exceptions during avro timestamp
> conversion.
> It is generally good idea to avoid throwing exceptions in performance
> critical sections, as exception creation is an expensive operation, and
> potentially repeating for many rows/values in a query can have drastic
> performance implications.
> Afaics there is no reason to convert numeric timestamp to a string and enter
> very lenient
> {{org.apache.hadoop.hive.common.type.TimestampTZUtil#parse(java.lang.String,
> java.time.ZoneId)}} to do timezone conversion.
> This patch changes the conversion of {{Date}} and {{Timestamp}} to
> {{TimestampTZ}} such that it doesn't invoke {{parse}}.
> JMH timings before:
> {code:java}
> Benchmark Mode Cnt Score Error
> Units
> TimestampTZUtilBench.convertDate avgt2 10091.990
> ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 10657.596
> ns/op
> {code}
> JMH timings after:
> {code:java}
> Benchmark Mode Cnt Score Error Units
> TimestampTZUtilBench.convertDate avgt2 48.371 ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 51.170 ns/op
> {code}
> JMH stack profile before:
> {code:java}
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertDate:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 97.4% 97.4% java.lang.Throwable.fillInStackTrace
> 1.6% 1.6% java.time.format.DateTimeFormatter.parse
> 0.2% 0.2% java.time.ZoneId.from
> 0.1% 0.1% java.util.HashMap.hash
> 0.1% 0.1% java.lang.Number.
> 0.1% 0.1%
> java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format
> 0.1% 0.1% java.lang.StringBuilder.append
> 0.1% 0.1% java.util.HashMap.putVal
> 0.1% 0.1% java.lang.String.valueOf
> 0.1% 0.1% java.util.regex.Pattern$BmpCharProperty.match
> 0.2% 0.2%
> ...
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertTimestamp:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 96.5% 96.5% java.lang.Throwable.fillInStackTrace
> 1.0% 1.0% java.time.format.DateTimeFormatter.parse
> 0.6% 0.6% org.apache.hadoop.hive.common.type.TimestampTZUtil.parse
> 0.4% 0.4%
[jira] [Commented] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972749#comment-16972749
]
David Mollitor commented on HIVE-16879:
---
[~abstractdog] And are you also able to check this one out? Thanks!
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-16879:
--
Status: Open (was: Patch Available)
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-16879:
--
Attachment: HIVE-16879.4.patch
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-16879) Review of AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-16879:
--
Status: Patch Available (was: Open)
> Review of AggregateStatsCache
> -
>
> Key: HIVE-16879
> URL: https://issues.apache.org/jira/browse/HIVE-16879
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-16879.1.patch, HIVE-16879.2.patch,
> HIVE-16879.3.patch, HIVE-16879.4.patch
>
>
> Improve cache key for cache implemented in
> {{org.apache.hadoop.hive.metastore.AggregateStatsCache}}.
> # Upgrade _debug_ logging to not generate text unless required
> # Changed _equals_ method to check first for the item most likely to be
> different, column name
> # Improve usage of Java 8 Map API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22484:
--
Status: Patch Available (was: Open)
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22484:
--
Attachment: HIVE-22484.1.patch
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22484.1.patch
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-22484:
-
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22477) Avro logical type timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-22477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972713#comment-16972713
]
Hive QA commented on HIVE-22477:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
17s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText
from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hive-jmh: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
28s{color} | {color:red} patch/itests/hive-jmh cannot run convertXmlToText from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19385/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19385/yetus/branch-findbugs-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19385/yetus/diff-checkstyle-itests_hive-jmh.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19385/yetus/patch-findbugs-itests_hive-jmh.txt
|
| modules | C: common itests/hive-jmh U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19385/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avro logical type timestamp conversion is slow
> --
>
> Key: HIVE-22477
> URL: https://issues.apache.org/jira/browse/HIVE-22477
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
> Environment: Hive 3.1.0
>Reporter: Sanjar Akhmedov
>Priority: Major
> Labels: Performance
> Attachments: HIVE-22477.1.patch, HIVE-22477.2.patch, flamegraph.svg
>
>
> We have an avro backed table with hundreds of billions timestamps. Simple
> {{SELECT COUNT(*) FROM t}} query takes many hours to complete in version
> 3.1.0 versus tens of minutes in version 1.2.1.
> Looking at the attached flamegraph of one of the yarn containers, hive
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Status: Patch Available (was: Open) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Status: Open (was: Patch Available) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22408: --- Attachment: HIVE-22408.4.patch > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch > > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22483) Vectorize UDF datetime_legacy_hybrid_calendar
[
https://issues.apache.org/jira/browse/HIVE-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972695#comment-16972695
]
Hive QA commented on HIVE-22483:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985626/HIVE-22483.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17709 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=300)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19384/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19384/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19384/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985626 - PreCommit-HIVE-Build
> Vectorize UDF datetime_legacy_hybrid_calendar
> -
>
> Key: HIVE-22483
> URL: https://issues.apache.org/jira/browse/HIVE-22483
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22483.01.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22483) Vectorize UDF datetime_legacy_hybrid_calendar
[
https://issues.apache.org/jira/browse/HIVE-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972672#comment-16972672
]
Hive QA commented on HIVE-22483:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 29 new + 33 unchanged - 4
fixed = 62 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
43s{color} | {color:red} ql generated 2 new + 1550 unchanged - 0 fixed = 1552
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Call to method of static java.text.DateFormat in
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDatetimeLegacyHybridCalendarDate.func(LongColumnVector,
LongColumnVector, int) At
VectorUDFDatetimeLegacyHybridCalendarDate.java:java.text.DateFormat in
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDatetimeLegacyHybridCalendarDate.func(LongColumnVector,
LongColumnVector, int) At
VectorUDFDatetimeLegacyHybridCalendarDate.java:[line 39] |
| | Call to method of static java.text.DateFormat in
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDatetimeLegacyHybridCalendarTimestamp.func(TimestampColumnVector,
TimestampColumnVector, int) At
VectorUDFDatetimeLegacyHybridCalendarTimestamp.java:java.text.DateFormat in
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDatetimeLegacyHybridCalendarTimestamp.func(TimestampColumnVector,
TimestampColumnVector, int) At
VectorUDFDatetimeLegacyHybridCalendarTimestamp.java:[line 37] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19384/dev-support/hive-personality.sh
|
| git revision | master / 9678ca1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19384/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19384/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19384/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19384/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorize UDF datetime_legacy_hybrid_calendar
> -
>
> Key: HIVE-22483
> URL: https://issues.apache.org/jira/browse/HIVE-22483
> Project: Hive
> Issue Type: Improvement
>
[jira] [Commented] (HIVE-21581) Remove Lock in GetInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972653#comment-16972653
]
David Mollitor commented on HIVE-21581:
---
[~dkuzmenko] Thank you for sharing the doc.
I assume you mean 'NN' or NameNode calls.
It is not clear to me why this lock exists. Perhaps you have better
understanding of the code?
It seems to me that the lock would be superfluous since there was a
compile-time lock. Each time this {{getInputSummary}} code gets called, there
is a new thread pool that gets created (default: 15 threads) and the work is
delegated to the pool. As I understand the code, the worst-case scenario would
be 15 threads running per query. Since the compile lock is now relaxed, we may
see more threads, and therefore more connections to the NameNode, but that is
to be expected, and preferred in order to increase query compilation time. We
can doc that the number of threads-per query should be lowered as the number of
concurrent compilations is increased.
More broadly, I had previously provided an implementation that would create a
single thread pool that all queries shared. [HIVE-19847] In this way, the
number of threads required to process a query does not automatically scale
linearly with the number of queries allowed to be compiled at once, but it
never gained any interest.
> Remove Lock in GetInputSummary
> --
>
> Key: HIVE-21581
> URL: https://issues.apache.org/jira/browse/HIVE-21581
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21581.1.patch, HIVE-21581.2.patch,
> HIVE-21581.3.patch
>
>
> Now that Hive compile lock has been relaxed in HIVE-20535, remove the
> {{getInputSummary}} lock:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2355]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972649#comment-16972649
]
Yongzhi Chen commented on HIVE-22461:
-
The failures are not related. Rebase and re-attach patch 4
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
>
[jira] [Updated] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-22461:
Attachment: HIVE-22461.4.patch
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch, HIVE-22461.4.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
>
[jira] [Updated] (HIVE-22461) NPE Metastore Transformer
[
https://issues.apache.org/jira/browse/HIVE-22461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-22461:
Attachment: (was: HIVE-22461.4.patch)
> NPE Metastore Transformer
> -
>
> Key: HIVE-22461
> URL: https://issues.apache.org/jira/browse/HIVE-22461
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-22461.1.patch, HIVE-22461.2.patch,
> HIVE-22461.3.patch
>
>
> The stack looks as following:
> {noformat}
> 2019-10-08 18:09:12,198 INFO
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer:
> [pool-6-thread-328]: Starting translation for processor
> Hiveserver2#3.1.2000.7.0.2.0...@vc0732.halxg.cloudera.com on list 1
> 2019-10-08 18:09:12,198 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-328]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16617)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-10-08 18:09:12,199 ERROR org.apache.thrift.server.TThreadPoolServer:
> [pool-6-thread-328]: Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transform(MetastoreDefaultTransformer.java:99)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:3391)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:3352)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2000.7.0.2.0-59.jar:3.1.2000.7.0.2.0-59]
> at com.sun.proxy.$Proxy28.get_table_req(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:16633)
>
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972645#comment-16972645
]
Hive QA commented on HIVE-22435:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985623/HIVE-22435.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17708 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.defaultConstraints
(batchId=224)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19383/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19383/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19383/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985623 - PreCommit-HIVE-Build
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
> [ERROR] Failures:
> [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed
> with error code = 2
> running
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> fname=limit_pushdown3.q
> See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log,
> or check ./ql/target/surefire-reports or
> ./itests/qtest/target/surefire-reports/ for specific test cases logs.
> org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed,
> vertexName=Reducer 2, vertexId=vertex_1572454329409_0001_9_01,
> diagnostics=[Task failed, taskId=task_1572454329409_0001_9_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1572454329409_0001_9_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field key from [0:key._col0,
> 1:key._col1]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: cannot find field key from
> [0:key._col0, 1:key._col1]
>
[jira] [Updated] (HIVE-21581) Remove Lock in GetInputSummary
[
https://issues.apache.org/jira/browse/HIVE-21581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21581:
--
Description:
Now that Hive compile lock has been relaxed in HIVE-20535, remove the
{{getInputSummary}} lock:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2355]
was:
Now that Hive compile lock has been relaxed in [HIVE-20535], remove the
{{getInputSummary}} lock:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2459]
> Remove Lock in GetInputSummary
> --
>
> Key: HIVE-21581
> URL: https://issues.apache.org/jira/browse/HIVE-21581
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21581.1.patch, HIVE-21581.2.patch,
> HIVE-21581.3.patch
>
>
> Now that Hive compile lock has been relaxed in HIVE-20535, remove the
> {{getInputSummary}} lock:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2355]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22479) Fix typo in GenericUDF
[ https://issues.apache.org/jira/browse/HIVE-22479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated HIVE-22479: --- Attachment: HIVE-22479.001.patch Status: Patch Available (was: Open) > Fix typo in GenericUDF > -- > > Key: HIVE-22479 > URL: https://issues.apache.org/jira/browse/HIVE-22479 > Project: Hive > Issue Type: Improvement >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: newbie, pull-request-available > Attachments: HIVE-22479.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > `Defered` corrected to `Deferred` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22475) Update slf4j to 1.7.25
[ https://issues.apache.org/jira/browse/HIVE-22475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972628#comment-16972628 ] Anishek Agarwal commented on HIVE-22475: [~sankarh] i thought you already had a change to move slf4j to the higher version ? > Update slf4j to 1.7.25 > -- > > Key: HIVE-22475 > URL: https://issues.apache.org/jira/browse/HIVE-22475 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > > Druid handler is already there. Updating will allow the entire project to be > on the same version. > https://github.com/apache/hive/blob/38190f3e95752c85188682d8a78d259455e173c2/itests/qtest-druid/pom.xml#L228 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22469: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972619#comment-16972619 ] David Mollitor commented on HIVE-22469: --- [~mgergely] Thank you for the review! Pushed to master. > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22469) Lower Metastore DB Connection Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22469: -- Fix Version/s: 4.0.0 > Lower Metastore DB Connection Pool Size in QTests > - > > Key: HIVE-22469 > URL: https://issues.apache.org/jira/browse/HIVE-22469 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch, > HIVE-22469.3.patch > > > Hive Metastore uses the 'HikariCP' database connection pool. The default > number of connections to the database is 10. For the Qtests, connecting to a > local DerbyDB, there need not be more than 4 connections. Anymore simply adds > undo overhead and by looking at the QTest logs, I see a bunch of 'connection > refused' from HikariCP. It may be the case that the standalone DB does not > support that many concurrent connections anyway. > [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] > I also unified all of the JDBC connection strings to ensure that each test is > using an in-memory database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22435) Exception when using VectorTopNKeyOperator operator
[
https://issues.apache.org/jira/browse/HIVE-22435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972604#comment-16972604
]
Hive QA commented on HIVE-22435:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 437 unchanged - 2
fixed = 437 total (was 439) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19383/dev-support/hive-personality.sh
|
| git revision | master / 5bbac26 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19383/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Exception when using VectorTopNKeyOperator operator
> ---
>
> Key: HIVE-22435
> URL: https://issues.apache.org/jira/browse/HIVE-22435
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-20150.15.patch, HIVE-22435.1.patch,
> HIVE-22435.2.patch, HIVE-22435.3.patch, HIVE-22435.4.patch
>
>
> Steps to reproduce:
> 1. Apply the attached patch
> {code}
> git apply -3 -p0 HIVE-20150.15.patch
> {code}
> 2. rebuild project
> {code:java}
> mvn clean install -DskipTests
> {code}
> 3. Run the following test
> {code:java}
> mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=limit_pushdown3.q -pl itests/qtest -Pitests
> {code}
> Query execution fails with exception
> {code:java}
> select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint
> order by ctinyint limit 20
> {code}
> {code:java}
>
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Status: Patch Available (was: Open) > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Status: Open (was: Patch Available) > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[ https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22464: -- Attachment: HIVE-22464.4.patch > Implement support for NULLS FIRST/LAST in TopNKeyOperator > - > > Key: HIVE-22464 > URL: https://issues.apache.org/jira/browse/HIVE-22464 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch, > HIVE-22464.3.patch, HIVE-22464.4.patch, HIVE-22464.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972567#comment-16972567
]
Hive QA commented on HIVE-22464:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12985621/HIVE-22464.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17707 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19382/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19382/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19382/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12985621 - PreCommit-HIVE-Build
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22453) Describe table unnecessarily fetches partitions
[ https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiko Uchida updated HIVE-22453: Attachment: HIVE-22453.2.patch > Describe table unnecessarily fetches partitions > --- > > Key: HIVE-22453 > URL: https://issues.apache.org/jira/browse/HIVE-22453 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2, 2.3.6 >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-22453.2.patch, HIVE-22453.patch > > > The simple describe table command without EXTENDED and FORMATTED (i.e., > DESCRIBE table_name) fetches all partitions when no partition is specified, > although it does not display partition statistics in nature. > The command should not fetch partitions since it can take a long time for a > large amount of partitions. > For instance, in our environment, the command takes around 8 seconds for a > table with 8760 (24 * 365) partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22411: Component/s: Transactions > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.2.patch, HIVE-22411.3.patch, Screen Shot > 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22411: Resolution: Fixed Status: Resolved (was: Patch Available) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.2.patch, HIVE-22411.3.patch, Screen Shot > 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972535#comment-16972535 ] László Bodor commented on HIVE-22411: - pushed to master, thanks [~amagyar] for the patch! > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.2.patch, HIVE-22411.3.patch, Screen Shot > 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22477) Avro logical type timestamp conversion is slow
[
https://issues.apache.org/jira/browse/HIVE-22477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjar Akhmedov updated HIVE-22477:
---
Attachment: HIVE-22477.2.patch
> Avro logical type timestamp conversion is slow
> --
>
> Key: HIVE-22477
> URL: https://issues.apache.org/jira/browse/HIVE-22477
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0
> Environment: Hive 3.1.0
>Reporter: Sanjar Akhmedov
>Priority: Major
> Labels: Performance
> Attachments: HIVE-22477.1.patch, HIVE-22477.2.patch, flamegraph.svg
>
>
> We have an avro backed table with hundreds of billions timestamps. Simple
> {{SELECT COUNT(*) FROM t}} query takes many hours to complete in version
> 3.1.0 versus tens of minutes in version 1.2.1.
> Looking at the attached flamegraph of one of the yarn containers, hive is
> spending most of the time throwing exceptions during avro timestamp
> conversion.
> It is generally good idea to avoid throwing exceptions in performance
> critical sections, as exception creation is an expensive operation, and
> potentially repeating for many rows/values in a query can have drastic
> performance implications.
> Afaics there is no reason to convert numeric timestamp to a string and enter
> very lenient
> {{org.apache.hadoop.hive.common.type.TimestampTZUtil#parse(java.lang.String,
> java.time.ZoneId)}} to do timezone conversion.
> This patch changes the conversion of {{Date}} and {{Timestamp}} to
> {{TimestampTZ}} such that it doesn't invoke {{parse}}.
> JMH timings before:
> {code:java}
> Benchmark Mode Cnt Score Error
> Units
> TimestampTZUtilBench.convertDate avgt2 10091.990
> ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 10657.596
> ns/op
> {code}
> JMH timings after:
> {code:java}
> Benchmark Mode Cnt Score Error Units
> TimestampTZUtilBench.convertDate avgt2 48.371 ns/op
> TimestampTZUtilBench.convertTimestamp avgt2 51.170 ns/op
> {code}
> JMH stack profile before:
> {code:java}
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertDate:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 97.4% 97.4% java.lang.Throwable.fillInStackTrace
> 1.6% 1.6% java.time.format.DateTimeFormatter.parse
> 0.2% 0.2% java.time.ZoneId.from
> 0.1% 0.1% java.util.HashMap.hash
> 0.1% 0.1% java.lang.Number.
> 0.1% 0.1%
> java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format
> 0.1% 0.1% java.lang.StringBuilder.append
> 0.1% 0.1% java.util.HashMap.putVal
> 0.1% 0.1% java.lang.String.valueOf
> 0.1% 0.1% java.util.regex.Pattern$BmpCharProperty.match
> 0.2% 0.2%
> ...
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertTimestamp:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 96.5% 96.5% java.lang.Throwable.fillInStackTrace
> 1.0% 1.0% java.time.format.DateTimeFormatter.parse
> 0.6% 0.6% org.apache.hadoop.hive.common.type.TimestampTZUtil.parse
> 0.4% 0.4% java.time.ZoneId.from
> 0.2% 0.2%
> java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format
> 0.2% 0.2% java.time.format.Parsed.resolveFields
> 0.2% 0.2% java.lang.String.valueOf
> 0.1% 0.1% java.lang.StringBuilder.append
> 0.1% 0.1% java.util.HashMap.hash
> 0.1% 0.1% java.time.format.DateTimeParseContext.toResolved
> 0.6% 0.6%
> {code}
> JMH stack profile after:
> {code:java}
> Secondary result
> "org.apache.hive.benchmark.common.TimestampTZUtilBench.convertDate:·stack":
> Stack profiler:
> [Thread state
> distributions]
> 100.0% RUNNABLE
> [Thread state:
> RUNNABLE]
> 91.6% 91.6% java.time.ZonedDateTime.ofInstant
> 8.0% 8.0%
> org.apache.hive.benchmark.common.generated.TimestampTZUtilBench_convertDate_jmhTest.convertDate_avgt_jmhStub
> 0.1% 0.1% java.time.zone.ZoneRules.
> 0.1% 0.1% java.time.LocalDateTime.ofEpochSecond
> 0.1% 0.1% org.apache.hadoop.hive.common.type.TimestampTZUtil.convert
> 0.1% 0.1% java.time.LocalDate.ofEpochDay
> 0.1% 0.1%
[jira] [Assigned] (HIVE-21266) Issue with single delta file
[
https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage reassigned HIVE-21266:
Assignee: Karen Coppage
> Issue with single delta file
>
>
> Key: HIVE-21266
> URL: https://issues.apache.org/jira/browse/HIVE-21266
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Karen Coppage
>Priority: Major
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1)
> {
> LOG.debug("Not compacting {}; current base is {} and there are {}
> deltas and {} originals", sd.getLocation(), dir
> .getBaseDirectory(), deltaCount, origCount);
> return;
> }
> {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where
> {{txnid:13}} was aborted. The code above will not rewrite the delta (which
> drops anything that belongs to the aborted txn) and transition the compaction
> to "ready_for_cleaning" state which will drop the metadata about the aborted
> txn in {{markCleaned()}}. Now aborted data will come back as committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22464) Implement support for NULLS FIRST/LAST in TopNKeyOperator
[
https://issues.apache.org/jira/browse/HIVE-22464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972514#comment-16972514
]
Hive QA commented on HIVE-22464:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} serde generated 0 new + 198 unchanged - 1 fixed =
198 total (was 199) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} ql generated 0 new + 1547 unchanged - 3 fixed = 1547
total (was 1550) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19382/dev-support/hive-personality.sh
|
| git revision | master / f361fc6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19382/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement support for NULLS FIRST/LAST in TopNKeyOperator
> -
>
> Key: HIVE-22464
> URL: https://issues.apache.org/jira/browse/HIVE-22464
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22464.1.patch, HIVE-22464.2.patch,
> HIVE-22464.3.patch, HIVE-22464.4.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972496#comment-16972496 ] David Mollitor commented on HIVE-22444: --- [~mgergely] Good question. I'm not sure how backwards compatible 3.3 is with the 2.x library currently in Hive. I want to introduce it to as few places as possible so that we don't end up testing with setups that don't reflect most production setups. (Do most setups have protobuf 3.x on the classpath?) I think the better option would be to open a separate ticket to upgrade the entire project to protobuf 3.x ([HIVE-20359]) > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.10.patch, > HIVE-22444.11.patch, HIVE-22444.2.patch, HIVE-22444.3.patch, > HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch, > HIVE-22444.7.patch, HIVE-22444.8.patch, HIVE-22444.9.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20256) Remove unused classes from Hive QL
[ https://issues.apache.org/jira/browse/HIVE-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972487#comment-16972487 ] David Mollitor commented on HIVE-20256: --- [~mgergely] Nice. You know I like this kind of cleaning work. :) >From my research, most of these classes were orphaned at some point when the >code relying on it changed to no longer include these classes. I did some google searches for the code usage more broadly. and there are some very limited uses of these classes out there. I verified that the patch does not modify any test classes, yet all the tests still pass. None of the classes being removed are annotated with any kind of API information. +1 on the patch, as long as it's applied only to Hive 4.x branch. There is no expectation of compatibility at this level (this patch introduced no API changes). I think the most polite thing to do would be to mark these classes as deprecated in the 3.x branch. > Remove unused classes from Hive QL > -- > > Key: HIVE-20256 > URL: https://issues.apache.org/jira/browse/HIVE-20256 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20256.01.patch, HIVE-20256.02.patch > > > Remove those classes from Hive QL which's name never appears anywhere else in > any apache project other than the class itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22483) Vectorize UDF datetime_legacy_hybrid_calendar
[ https://issues.apache.org/jira/browse/HIVE-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22483: - Attachment: HIVE-22483.01.patch Status: Patch Available (was: Open) > Vectorize UDF datetime_legacy_hybrid_calendar > - > > Key: HIVE-22483 > URL: https://issues.apache.org/jira/browse/HIVE-22483 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22483.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
