[jira] [Updated] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-13901: Attachment: HIVE-13901.8.patch > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch, HIVE-13901.8.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[ https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367175#comment-15367175 ] Rajesh Balamohan commented on HIVE-13901: - Thanks [~ashutoshc]; Was able to reproduce the issue you encountered. I have updated the patch in RB https://reviews.apache.org/r/48159/ > Hivemetastore add partitions can be slow depending on filesystems > - > > Key: HIVE-13901 > URL: https://issues.apache.org/jira/browse/HIVE-13901 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch, > HIVE-13901.6.patch, HIVE-13901.7.patch > > > Depending on FS, creating external tables & adding partitions can be > expensive (e.g msck which adds all partitions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14173) NPE was thrown after enabling directsql in the middle of session
[
https://issues.apache.org/jira/browse/HIVE-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367173#comment-15367173
]
Hive QA commented on HIVE-14173:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816750/HIVE-14173.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/419/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/419/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-419/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
java.io.IOException: Could not create
/data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-419/succeeded/TestMinimrCliDriver-load_fs2.q-bucketmapjoin7.q-quotedid_smb.q-and-1-more
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816750 - PreCommit-HIVE-MASTER-Build
> NPE was thrown after enabling directsql in the middle of session
>
>
> Key: HIVE-14173
> URL: https://issues.apache.org/jira/browse/HIVE-14173
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-14173.patch, HIVE-14173.patch
>
>
> hive.metastore.try.direct.sql is initially set to false in HMS hive-site.xml,
> then changed to true using set metaconf command in the middle of a session,
> running a query will be thrown NPE with error message is as following:
> {code}
> 2016-07-06T17:44:41,489 ERROR [pool-5-thread-2]: metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invokeInternal(192)) -
> MetaException(message:java.lang.NullPointerException)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5741)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rethrowException(HiveMetaStore.java:4771)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy18.get_partitions_by_expr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12048)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12032)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.(ObjectStore.java:2667)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetListHelper.(ObjectStore.java:2825)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$4.(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:2400)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Resolved] (HIVE-14183) hive 2.1.0 beeline cannot show verbose log
[ https://issues.apache.org/jira/browse/HIVE-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack resolved HIVE-14183. - Resolution: Not A Bug Release Note: set hive.async.log.enabled=false; the verbose log can show normally > hive 2.1.0 beeline cannot show verbose log > -- > > Key: HIVE-14183 > URL: https://issues.apache.org/jira/browse/HIVE-14183 > Project: Hive > Issue Type: Bug > Components: Beeline, HiveServer2 >Affects Versions: 2.1.0 >Reporter: jack >Assignee: jack > > In Hive 1.2.1 , config the log verbose , the beeline client will show the > verbose log like this: > 16/07/07 13:29:33 INFO mapreduce.Job: The url to track the job: > http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Starting Job = job_1467708727273_0035, > Tracking URL = http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Kill Command = > /hadoop/hadoop-2.7.2/bin/hadoop job -kill job_1467708727273_0035 > 16/07/07 13:30:07 INFO exec.Task: Hadoop job information for Stage-1: number > of mappers: 1; number of reducers: 1 > 16/07/07 13:30:07 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 16/07/07 13:30:07 INFO exec.Task: 2016-07-07 13:30:07,905 Stage-1 map = 0%, > reduce = 0% > 16/07/07 13:30:17 INFO exec.Task: 2016-07-07 13:30:17,757 Stage-1 map = 100%, > reduce = 0%, Cumulative CPU 2.39 sec > 16/07/07 13:30:28 > the verbose config: > > hive.server2.logging.operation.enabled > true > When true, HS2 will save operation logs and make them > available for clients > > > hive.server2.logging.operation.log.location > /hadooplog/apache-hive-1.2.1-bin/operation_logs > Top level directory where operation logs are stored if > logging functionality is enabled > > > hive.server2.logging.operation.level > VERBOSE > > Expects one of [none, execution, performance, verbose]. > HS2 operation logging mode available to clients to be set at session > level. > For this to work, hive.server2.logging.operation.enabled should be set > to true. > NONE: Ignore any logging > EXECUTION: Log completion of tasks > PERFORMANCE: Execution + Performance logs > VERBOSE: All logs > > > However, when upgrade to hive 2.1.0, with the same verbose configure, the > verbose log cannot show. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14183) hive 2.1.0 beeline cannot show verbose log
[ https://issues.apache.org/jira/browse/HIVE-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367162#comment-15367162 ] jack commented on HIVE-14183: - set hive.async.log.enabled=false; can resolve this issue. It's not a bug. > hive 2.1.0 beeline cannot show verbose log > -- > > Key: HIVE-14183 > URL: https://issues.apache.org/jira/browse/HIVE-14183 > Project: Hive > Issue Type: Bug > Components: Beeline, HiveServer2 >Affects Versions: 2.1.0 >Reporter: jack > > In Hive 1.2.1 , config the log verbose , the beeline client will show the > verbose log like this: > 16/07/07 13:29:33 INFO mapreduce.Job: The url to track the job: > http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Starting Job = job_1467708727273_0035, > Tracking URL = http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Kill Command = > /hadoop/hadoop-2.7.2/bin/hadoop job -kill job_1467708727273_0035 > 16/07/07 13:30:07 INFO exec.Task: Hadoop job information for Stage-1: number > of mappers: 1; number of reducers: 1 > 16/07/07 13:30:07 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 16/07/07 13:30:07 INFO exec.Task: 2016-07-07 13:30:07,905 Stage-1 map = 0%, > reduce = 0% > 16/07/07 13:30:17 INFO exec.Task: 2016-07-07 13:30:17,757 Stage-1 map = 100%, > reduce = 0%, Cumulative CPU 2.39 sec > 16/07/07 13:30:28 > the verbose config: > > hive.server2.logging.operation.enabled > true > When true, HS2 will save operation logs and make them > available for clients > > > hive.server2.logging.operation.log.location > /hadooplog/apache-hive-1.2.1-bin/operation_logs > Top level directory where operation logs are stored if > logging functionality is enabled > > > hive.server2.logging.operation.level > VERBOSE > > Expects one of [none, execution, performance, verbose]. > HS2 operation logging mode available to clients to be set at session > level. > For this to work, hive.server2.logging.operation.enabled should be set > to true. > NONE: Ignore any logging > EXECUTION: Log completion of tasks > PERFORMANCE: Execution + Performance logs > VERBOSE: All logs > > > However, when upgrade to hive 2.1.0, with the same verbose configure, the > verbose log cannot show. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14183) hive 2.1.0 beeline cannot show verbose log
[ https://issues.apache.org/jira/browse/HIVE-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack reassigned HIVE-14183: --- Assignee: jack > hive 2.1.0 beeline cannot show verbose log > -- > > Key: HIVE-14183 > URL: https://issues.apache.org/jira/browse/HIVE-14183 > Project: Hive > Issue Type: Bug > Components: Beeline, HiveServer2 >Affects Versions: 2.1.0 >Reporter: jack >Assignee: jack > > In Hive 1.2.1 , config the log verbose , the beeline client will show the > verbose log like this: > 16/07/07 13:29:33 INFO mapreduce.Job: The url to track the job: > http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Starting Job = job_1467708727273_0035, > Tracking URL = http://host:8088/proxy/application_1467708727273_0035/ > 16/07/07 13:29:33 INFO exec.Task: Kill Command = > /hadoop/hadoop-2.7.2/bin/hadoop job -kill job_1467708727273_0035 > 16/07/07 13:30:07 INFO exec.Task: Hadoop job information for Stage-1: number > of mappers: 1; number of reducers: 1 > 16/07/07 13:30:07 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 16/07/07 13:30:07 INFO exec.Task: 2016-07-07 13:30:07,905 Stage-1 map = 0%, > reduce = 0% > 16/07/07 13:30:17 INFO exec.Task: 2016-07-07 13:30:17,757 Stage-1 map = 100%, > reduce = 0%, Cumulative CPU 2.39 sec > 16/07/07 13:30:28 > the verbose config: > > hive.server2.logging.operation.enabled > true > When true, HS2 will save operation logs and make them > available for clients > > > hive.server2.logging.operation.log.location > /hadooplog/apache-hive-1.2.1-bin/operation_logs > Top level directory where operation logs are stored if > logging functionality is enabled > > > hive.server2.logging.operation.level > VERBOSE > > Expects one of [none, execution, performance, verbose]. > HS2 operation logging mode available to clients to be set at session > level. > For this to work, hive.server2.logging.operation.enabled should be set > to true. > NONE: Ignore any logging > EXECUTION: Log completion of tasks > PERFORMANCE: Execution + Performance logs > VERBOSE: All logs > > > However, when upgrade to hive 2.1.0, with the same verbose configure, the > verbose log cannot show. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: HIVE-13934.6.patch > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: (was: HIVE-13934.6.patch) > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13685) Vectorization: Date*ColScalar parses the scalar in every loop
[
https://issues.apache.org/jira/browse/HIVE-13685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13685:
---
Status: Patch Available (was: Open)
> Vectorization: Date*ColScalar parses the scalar in every loop
> -
>
> Key: HIVE-13685
> URL: https://issues.apache.org/jira/browse/HIVE-13685
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13685.1.patch, HIVE-13685.2.patch,
> HIVE-13685.WIP.patch
>
>
> {code}
> @Override
> public void evaluate(VectorizedRowBatch batch) {
> ...
> switch (inputTypes[0]) {
> ...
> case STRING:
> case CHAR:
> case VARCHAR:
> try {
> baseDate = formatter.parse(new String(stringValue, "UTF-8"));
> break;
> {code}
> This is parsing the constant scalar for every evaluate() call.
> Parsing the constant in the initialize op will save thousands of expensive
> calls into Calendar operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13685) Vectorization: Date*ColScalar parses the scalar in every loop
[
https://issues.apache.org/jira/browse/HIVE-13685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13685:
---
Attachment: HIVE-13685.2.patch
Fix for ColScalar and ScalarCol impls.
They really shouldn't be twice the impl, at least for the date_diff() impl. The
date_add() is pretty clear why.
> Vectorization: Date*ColScalar parses the scalar in every loop
> -
>
> Key: HIVE-13685
> URL: https://issues.apache.org/jira/browse/HIVE-13685
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13685.1.patch, HIVE-13685.2.patch,
> HIVE-13685.WIP.patch
>
>
> {code}
> @Override
> public void evaluate(VectorizedRowBatch batch) {
> ...
> switch (inputTypes[0]) {
> ...
> case STRING:
> case CHAR:
> case VARCHAR:
> try {
> baseDate = formatter.parse(new String(stringValue, "UTF-8"));
> break;
> {code}
> This is parsing the constant scalar for every evaluate() call.
> Parsing the constant in the initialize op will save thousands of expensive
> calls into Calendar operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14170:
Attachment: HIVE-14170.1.patch
> Beeline IncrementalRows should buffer rows and incrementally re-calculate
> width if TableOutputFormat is used
>
>
> Key: HIVE-14170
> URL: https://issues.apache.org/jira/browse/HIVE-14170
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14170.1.patch
>
>
> If {{--incremental}} is specified in Beeline, rows are meant to be printed
> out immediately. However, if {{TableOutputFormat}} is used with this option
> the formatting can look really off.
> The reason is that {{IncrementalRows}} does not do a global calculation of
> the optimal width size for {{TableOutputFormat}} (it can't because it only
> sees one row at a time). The output of {{BufferedRows}} looks much better
> because it can do this global calculation.
> If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width
> should be re-calculated every "x" rows ("x" can be configurable and by
> default it can be 1000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14173) NPE was thrown after enabling directsql in the middle of session
[
https://issues.apache.org/jira/browse/HIVE-14173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-14173:
---
Attachment: HIVE-14173.patch
Resubmit the patch to kick off the precommit build
> NPE was thrown after enabling directsql in the middle of session
>
>
> Key: HIVE-14173
> URL: https://issues.apache.org/jira/browse/HIVE-14173
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-14173.patch, HIVE-14173.patch
>
>
> hive.metastore.try.direct.sql is initially set to false in HMS hive-site.xml,
> then changed to true using set metaconf command in the middle of a session,
> running a query will be thrown NPE with error message is as following:
> {code}
> 2016-07-06T17:44:41,489 ERROR [pool-5-thread-2]: metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invokeInternal(192)) -
> MetaException(message:java.lang.NullPointerException)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5741)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rethrowException(HiveMetaStore.java:4771)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy18.get_partitions_by_expr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12048)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiveMetastore.java:12032)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.(ObjectStore.java:2667)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetListHelper.(ObjectStore.java:2825)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$4.(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:2410)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:2400)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy17.getPartitionsByExpr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4749)
> ... 20 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13685) Vectorization: Date*ColScalar parses the scalar in every loop
[
https://issues.apache.org/jira/browse/HIVE-13685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13685:
---
Attachment: HIVE-13685.1.patch
> Vectorization: Date*ColScalar parses the scalar in every loop
> -
>
> Key: HIVE-13685
> URL: https://issues.apache.org/jira/browse/HIVE-13685
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13685.1.patch, HIVE-13685.WIP.patch
>
>
> {code}
> @Override
> public void evaluate(VectorizedRowBatch batch) {
> ...
> switch (inputTypes[0]) {
> ...
> case STRING:
> case CHAR:
> case VARCHAR:
> try {
> baseDate = formatter.parse(new String(stringValue, "UTF-8"));
> break;
> {code}
> This is parsing the constant scalar for every evaluate() call.
> Parsing the constant in the initialize op will save thousands of expensive
> calls into Calendar operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367083#comment-15367083 ] Ferdinand Xu commented on HIVE-14191: - Please update the status of this JIRA to package-available to trigger the precommit. You may do it later since precommit is broken. > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.1.0 >Reporter: Ziyang Zhao >Assignee: Ziyang Zhao > Attachments: HIVE-14191.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13873) Column pruning for nested fields
[ https://issues.apache.org/jira/browse/HIVE-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-13873: Attachment: HIVE-13873.wip.patch As talked offline with [~xuefuz], the first patch will resolve nested struct without complex group like list, map and array. > Column pruning for nested fields > > > Key: HIVE-13873 > URL: https://issues.apache.org/jira/browse/HIVE-13873 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > Attachments: HIVE-13873.wip.patch > > > Some columnar file formats such as Parquet store fields in struct type also > column by column using encoding described in Google Dramel pager. It's very > common in big data where data are stored in structs while queries only needs > a subset of the the fields in the structs. However, presently Hive still > needs to read the whole struct regardless whether all fields are selected. > Therefore, pruning unwanted sub-fields in struct or nested fields at file > reading time would be a big performance boost for such scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13932) Hive SMB Map Join with small set of LIMIT failed with NPE
[ https://issues.apache.org/jira/browse/HIVE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13932: - Fix Version/s: 2.1.1 > Hive SMB Map Join with small set of LIMIT failed with NPE > - > > Key: HIVE-13932 > URL: https://issues.apache.org/jira/browse/HIVE-13932 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13932.1.patch > > > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c limit 1; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13932) Hive SMB Map Join with small set of LIMIT failed with NPE
[ https://issues.apache.org/jira/browse/HIVE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367055#comment-15367055 ] Prasanth Jayachandran commented on HIVE-13932: -- Committed this patch to branch-2.1 > Hive SMB Map Join with small set of LIMIT failed with NPE > - > > Key: HIVE-13932 > URL: https://issues.apache.org/jira/browse/HIVE-13932 > Project: Hive > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13932.1.patch > > > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c limit 1; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13878: --- Status: Patch Available (was: Open) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13878: --- Attachment: HIVE-13878.05.patch > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13878: --- Status: Open (was: Patch Available) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367025#comment-15367025
]
Hive QA commented on HIVE-14192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816729/HIVE-14192.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/417/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/417/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-417/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Excluding org.apache.tez:tez-mapreduce:jar:0.8.3 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.6.0 from
the shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-core_2.10:jar:1.6.0 from the shaded jar.
[INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from the shaded jar.
[INFO] Excluding com.twitter:chill-java:jar:0.5.0 from the shaded jar.
[INFO] Excluding org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.6.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.1 from
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.1
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.1
from the shaded jar.
[INFO] Excluding org.apache.spark:spark-launcher_2.10:jar:1.6.0 from the shaded
jar.
[INFO] Excluding org.apache.spark:spark-network-common_2.10:jar:1.6.0 from the
shaded jar.
[INFO] Excluding org.apache.spark:spark-network-shuffle_2.10:jar:1.6.0 from the
shaded jar.
[INFO] Excluding org.apache.spark:spark-unsafe_2.10:jar:1.6.0 from the shaded
jar.
[INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from the shaded jar.
[INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from the shaded jar.
[INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the shaded jar.
[INFO] Excluding com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded
jar.
[INFO] Excluding com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded
jar.
[INFO] Excluding com.typesafe:config:jar:1.2.1 from the shaded jar.
[INFO] Excluding org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded
jar.
[INFO] Excluding com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded
jar.
[INFO] Excluding org.scala-lang:scala-library:jar:2.10.4 from the shaded jar.
[INFO] Excluding org.json4s:json4s-jackson_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.json4s:json4s-core_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.json4s:json4s-ast_2.10:jar:3.2.10 from the shaded jar.
[INFO] Excluding org.scala-lang:scalap:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar.
[INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1 from the
shaded jar.
[INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-graphite:jar:3.1.2 from the
shaded jar.
[INFO] Excluding
com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.4 from the
shaded jar.
[INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.4 from the shaded jar.
[INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-client:jar:0.8.2 from the shaded
jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-hdfs:jar:0.8.2 from the
shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-s3:jar:0.8.2 from the
shaded jar.
[INFO] Excluding org.tachyonproject:tachyon-underfs-local:jar:0.8.2 from the
shaded jar.
[INFO] Excluding net.razorvine:pyrolite:jar:4.9 from the shaded jar.
[INFO] Excluding net.sf.py4j:py4j:jar:0.9 from the shaded jar.
[INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.10 from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing
/data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar
with
/data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at:
/data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at:
/data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml
[jira] [Updated] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-14192:
--
Status: Patch Available (was: Open)
> False positive error due to thrift
> --
>
> Key: HIVE-14192
> URL: https://issues.apache.org/jira/browse/HIVE-14192
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 2.1.0, 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14192.patch
>
>
> Given Thrift definition like this
> {noformat}
> struct LockComponent {
> 1: required LockType type,
> 2: required LockLevel level,
> 3: required string dbname,
> 4: optional string tablename,
> 5: optional string partitionname,
> 6: optional DataOperationType operationType = DataOperationType.UNSET,
> 7: optional bool isAcid = false
> }
> {noformat}
> The generated LockComponent has
> {noformat}
> public LockComponent() {
> this.operationType =
> org.apache.hadoop.hive.metastore.api.DataOperationType.UNSET;
> this.isAcid = false;
> }
> public boolean isSetOperationType() {
> return this.operationType != null;
> }
> public boolean isSetIsAcid() {
> return EncodingUtils.testBit(__isset_bitfield, __ISACID_ISSET_ID);
> }
> {noformat}
> So bottom line is even if LockComponent is created by old version of the
> client which doesn't have operationType filed, isSetOperationType() will
> still return true on the server.
> This causes a false positive exception in TxnHandler.enqueueLockWithRetry()
> during Rolling Upgrade scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14192) False positive error due to thrift
[
https://issues.apache.org/jira/browse/HIVE-14192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-14192:
--
Attachment: HIVE-14192.patch
> False positive error due to thrift
> --
>
> Key: HIVE-14192
> URL: https://issues.apache.org/jira/browse/HIVE-14192
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14192.patch
>
>
> Given Thrift definition like this
> {noformat}
> struct LockComponent {
> 1: required LockType type,
> 2: required LockLevel level,
> 3: required string dbname,
> 4: optional string tablename,
> 5: optional string partitionname,
> 6: optional DataOperationType operationType = DataOperationType.UNSET,
> 7: optional bool isAcid = false
> }
> {noformat}
> The generated LockComponent has
> {noformat}
> public LockComponent() {
> this.operationType =
> org.apache.hadoop.hive.metastore.api.DataOperationType.UNSET;
> this.isAcid = false;
> }
> public boolean isSetOperationType() {
> return this.operationType != null;
> }
> public boolean isSetIsAcid() {
> return EncodingUtils.testBit(__isset_bitfield, __ISACID_ISSET_ID);
> }
> {noformat}
> So bottom line is even if LockComponent is created by old version of the
> client which doesn't have operationType filed, isSetOperationType() will
> still return true on the server.
> This causes a false positive exception in TxnHandler.enqueueLockWithRetry()
> during Rolling Upgrade scenarios.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367013#comment-15367013 ] Ashutosh Chauhan commented on HIVE-13930: - Is there a spark jar without hadoop that we can use? If not, I don't see any solution other than disabling spark tests. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14188) LLAPIF: wrong user field is used from the token
[
https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366998#comment-15366998
]
Hive QA commented on HIVE-14188:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816681/HIVE-14188.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/416/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/416/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-416/
Messages:
{noformat}
This message was trimmed, see log for full details
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-constants/0.9.18/shared-ldap-constants-0.9.18.pom
(2 KB at 60.1 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/shared/shared-cursor/0.9.18/shared-cursor-0.9.18.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-cursor/0.9.18/shared-cursor-0.9.18.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-cursor/0.9.18/shared-cursor-0.9.18.pom
(2 KB at 18.4 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/shared/shared-ldap-schema/0.9.18/shared-ldap-schema-0.9.18.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema/0.9.18/shared-ldap-schema-0.9.18.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema/0.9.18/shared-ldap-schema-0.9.18.pom
(3 KB at 97.7 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/shared/shared-ldif/0.9.18/shared-ldif-0.9.18.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldif/0.9.18/shared-ldif-0.9.18.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldif/0.9.18/shared-ldif-0.9.18.pom
(4 KB at 108.6 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/shared/shared-ldap-schema-manager/0.9.18/shared-ldap-schema-manager-0.9.18.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema-manager/0.9.18/shared-ldap-schema-manager-0.9.18.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema-manager/0.9.18/shared-ldap-schema-manager-0.9.18.pom
(3 KB at 27.0 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/shared/shared-ldap-schema-loader/0.9.18/shared-ldap-schema-loader-0.9.18.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema-loader/0.9.18/shared-ldap-schema-loader-0.9.18.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/shared/shared-ldap-schema-loader/0.9.18/shared-ldap-schema-loader-0.9.18.pom
(3 KB at 91.5 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/mina/mina-core/2.0.0-RC1/mina-core-2.0.0-RC1.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/mina/mina-core/2.0.0-RC1/mina-core-2.0.0-RC1.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/mina/mina-core/2.0.0-RC1/mina-core-2.0.0-RC1.pom
(2 KB at 58.5 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/mina/mina-parent/2.0.0-RC1/mina-parent-2.0.0-RC1.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/mina/mina-parent/2.0.0-RC1/mina-parent-2.0.0-RC1.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/mina/mina-parent/2.0.0-RC1/mina-parent-2.0.0-RC1.pom
(10 KB at 341.3 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/mina/build/2.0.0-RC1/build-2.0.0-RC1.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/mina/build/2.0.0-RC1/build-2.0.0-RC1.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/mina/build/2.0.0-RC1/build-2.0.0-RC1.pom
(8 KB at 270.6 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/server/apacheds-server-integ/1.5.6/apacheds-server-integ-1.5.6.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-server-integ/1.5.6/apacheds-server-integ-1.5.6.pom
Downloaded:
http://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-server-integ/1.5.6/apacheds-server-integ-1.5.6.pom
(4 KB at 116.0 KB/sec)
Downloading:
http://www.datanucleus.org/downloads/maven2/org/apache/directory/server/apacheds-parent/1.5.6/apacheds-parent-1.5.6.pom
Downloading:
http://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-parent/1.5.6/apacheds-parent-1.5.6.pom
Downloaded:
[jira] [Commented] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[
https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366989#comment-15366989
]
Hive QA commented on HIVE-13934:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816726/HIVE-13934.6.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/415/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/415/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-415/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-415/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f40ea94 HIVE-14132 : Don't fail config validation for removed
configs (Ashutosh Chauhan via Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f40ea94 HIVE-14132 : Don't fail config validation for removed
configs (Ashutosh Chauhan via Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
patching file common/src/java/org/apache/hive/common/util/HiveStringUtils.java
patching file
common/src/test/org/apache/hive/common/util/TestHiveStringUtils.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hive-ptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests
-Dmaven.repo.local=/data/hive-ptest/working/maven
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/java/jdk1.8.0_25/bin/java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816726 - PreCommit-HIVE-MASTER-Build
> Configure Tez to make nocondiional task size memory available for the
> Processor
> ---
>
> Key: HIVE-13934
> URL: https://issues.apache.org/jira/browse/HIVE-13934
> Project: Hive
> Issue Type: Bug
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch,
> HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch
>
>
> Currently, noconditionaltasksize is not validated against the container size,
> the reservations made in the container by Tez for Inputs / Outputs etc.
> Check this at compile time to see if enough memory is available, or set up
> the vertex to reserve additional memory for the Processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14188) LLAPIF: wrong user field is used from the token
[
https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366987#comment-15366987
]
Hive QA commented on HIVE-14188:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816681/HIVE-14188.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/414/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/414/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-414/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-414/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source ]]
+ git clone https://github.com/apache/hive.git apache-github-source-source
Cloning into 'apache-github-source-source'...
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f40ea94 HIVE-14132 : Don't fail config validation for removed
configs (Ashutosh Chauhan via Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f40ea94 HIVE-14132 : Don't fail config validation for removed
configs (Ashutosh Chauhan via Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hive-ptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests
-Dmaven.repo.local=/data/hive-ptest/working/maven
Error: JAVA_HOME is not defined correctly.
We cannot execute /usr/java/jdk1.8.0_25/bin/java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816681 - PreCommit-HIVE-MASTER-Build
> LLAPIF: wrong user field is used from the token
> ---
>
> Key: HIVE-14188
> URL: https://issues.apache.org/jira/browse/HIVE-14188
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14188.patch
>
>
> realUser is not usually set in all cases for delegation tokens; we should use
> the owner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13934: - Attachment: HIVE-13934.6.patch > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14039) HiveServer2: Make the usage of server with JDBC thirft serde enabled, backward compatible for older clients
[ https://issues.apache.org/jira/browse/HIVE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14039: Assignee: Ziyang Zhao > HiveServer2: Make the usage of server with JDBC thirft serde enabled, > backward compatible for older clients > --- > > Key: HIVE-14039 > URL: https://issues.apache.org/jira/browse/HIVE-14039 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.1 >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14191: Assignee: Ziyang Zhao > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.1.0 >Reporter: Ziyang Zhao >Assignee: Ziyang Zhao > Attachments: HIVE-14191.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14185) Join query fails if the left table is empty and where condition searches in a list containing null
[
https://issues.apache.org/jira/browse/HIVE-14185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366959#comment-15366959
]
Gopal V commented on HIVE-14185:
[~fmantlik]: because IN doesn't work the way you want.
col2 IN (null, 'A') looks like (col2 == null or col2 == 'A')
nothing is equal to null, even null itself, so it folds into (col2 == 'A'),
which implicitly does 'IS NOT NULL' internally.
If your goal is to look for nulls, equality is the wrong expression - for
example, you can filter nulls out by doing col2 == col2, which works like "IS
NOT NULL".
> Join query fails if the left table is empty and where condition searches in a
> list containing null
> --
>
> Key: HIVE-14185
> URL: https://issues.apache.org/jira/browse/HIVE-14185
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Frantisek Mantlik
>
> If the table on the left hand side of the join is empty and the right hand
> side table contains data, the following query fails:
> {{SELECT * FROM left JOIN right ON left.col1=right.col1 WHERE left.col2 IN
> (null, 'A')}}
> while the following query runs without problems:
> {{SELECT * FROM left JOIN right ON left.col1=right.col1 WHERE (left.col2 IS
> NULL or left.col2 IN ('A'))}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14191: Affects Version/s: 2.1.0 > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.1.0 >Reporter: Ziyang Zhao > Attachments: HIVE-14191.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14191: Component/s: JDBC HiveServer2 > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.1.0 >Reporter: Ziyang Zhao > Attachments: HIVE-14191.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12427) HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-12427: --- Assignee: (was: Ziyang Zhao) > HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance > > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta > > The current implementation does a lot of CPU intensive work in HiveServer2. > It deserializes the temporary results written to disk and also serializes > the results into thrift objects in HS2. This adds to the latency of fetching > results via HS2. Doing that work in the task nodes would get that work done > in parallel and scale it, thereby reducing the time it takes to retrieve > large results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-14191: --- Attachment: HIVE-14191.1.patch Create a new api version > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task >Reporter: Ziyang Zhao > Attachments: HIVE-14191.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12427) HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao reassigned HIVE-12427: -- Assignee: Ziyang Zhao > HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance > > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta >Assignee: Ziyang Zhao > > The current implementation does a lot of CPU intensive work in HiveServer2. > It deserializes the temporary results written to disk and also serializes > the results into thrift objects in HS2. This adds to the latency of fetching > results via HS2. Doing that work in the task nodes would get that work done > in parallel and scale it, thereby reducing the time it takes to retrieve > large results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14191) bump a new api version for ThriftJDBCBinarySerde changes
[ https://issues.apache.org/jira/browse/HIVE-14191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ziyang Zhao updated HIVE-14191: --- Issue Type: Sub-task (was: Bug) Parent: HIVE-12427 > bump a new api version for ThriftJDBCBinarySerde changes > > > Key: HIVE-14191 > URL: https://issues.apache.org/jira/browse/HIVE-14191 > Project: Hive > Issue Type: Sub-task >Reporter: Ziyang Zhao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14133) Don't fail config validation for removed configs
[ https://issues.apache.org/jira/browse/HIVE-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned HIVE-14133: --- Assignee: Ashutosh Chauhan > Don't fail config validation for removed configs > > > Key: HIVE-14133 > URL: https://issues.apache.org/jira/browse/HIVE-14133 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > Users may have set config in their scripts. If we remove said config in later > version then config validation code will throw exception for scripts > containing said config. This unnecessary incompatibility can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14133) Don't fail config validation for removed configs
[ https://issues.apache.org/jira/browse/HIVE-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-14133. - Resolution: Duplicate > Don't fail config validation for removed configs > > > Key: HIVE-14133 > URL: https://issues.apache.org/jira/browse/HIVE-14133 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > > Users may have set config in their scripts. If we remove said config in later > version then config validation code will throw exception for scripts > containing said config. This unnecessary incompatibility can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14132) Don't fail config validation for removed configs
[ https://issues.apache.org/jira/browse/HIVE-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14132: Resolution: Fixed Assignee: Ashutosh Chauhan Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Not sure how to enforce that. Pushed to master & branch-2.1 > Don't fail config validation for removed configs > > > Key: HIVE-14132 > URL: https://issues.apache.org/jira/browse/HIVE-14132 > Project: Hive > Issue Type: Bug > Components: Configuration >Affects Versions: 2.0.0, 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14132.patch > > > Users may have set config in their scripts. If we remove said config in later > version then config validation code will throw exception for scripts > containing said config. This unnecessary incompatibility can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde
[
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366890#comment-15366890
]
Vaibhav Gumashta commented on HIVE-13965:
-
[~ziyangz] Thanks for the analysis. I'll commit shortly.
> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-13965.1.patch.txt
>
>
> This error can be reproduced by enabling thrift binary serde, using beeline
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java.
> If the resultset is empty, function process() will not be called, so variable
> "fpaths" will not be set. When run into CloseOp(),
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
> HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS)
> &&
>
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
> {
> try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
> } catch (SerDeException | IOException e) {
>throw new HiveException(e);
> }
> }
> Here fpaths is null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366889#comment-15366889
]
Vaibhav Gumashta commented on HIVE-13723:
-
+1 pending tests.
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
> Attachments: HIVE-13723.2.patch.txt, HIVE-13723.3.patch
>
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-14189) backport HIVE-13945 to branch-1
[ https://issues.apache.org/jira/browse/HIVE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366845#comment-15366845 ] Sergey Shelukhin commented on HIVE-14189: - [~ashutoshc] fyi > backport HIVE-13945 to branch-1 > --- > > Key: HIVE-14189 > URL: https://issues.apache.org/jira/browse/HIVE-14189 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14189.01-branch-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14189) backport HIVE-13945 to branch-1
[ https://issues.apache.org/jira/browse/HIVE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14189: Attachment: HIVE-14189.01-branch-1.patch branch-1 patch. More or less equivalent, with one significant change - UDFLeast behavior was change similarly to HIVE-12082 to allow the conversion of different decimals (which also introduced the behavior from HIVE-12082 with regard to nulls). > backport HIVE-13945 to branch-1 > --- > > Key: HIVE-14189 > URL: https://issues.apache.org/jira/browse/HIVE-14189 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14189.01-branch-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14189) backport HIVE-13945 to branch-1
[ https://issues.apache.org/jira/browse/HIVE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14189: Target Version/s: 1.3.0 Status: Patch Available (was: Open) > backport HIVE-13945 to branch-1 > --- > > Key: HIVE-14189 > URL: https://issues.apache.org/jira/browse/HIVE-14189 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14189.01-branch-1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12082) Null comparison for greatest and least operator
[
https://issues.apache.org/jira/browse/HIVE-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366842#comment-15366842
]
Sergey Shelukhin commented on HIVE-12082:
-
This behavior will be propagated to branch-1 as part of HIVE-14189. Not sure if
I should add to fixVersion since I'm not committing the actual patch...
> Null comparison for greatest and least operator
> ---
>
> Key: HIVE-12082
> URL: https://issues.apache.org/jira/browse/HIVE-12082
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: 2.0.0
>
> Attachments: HIVE-12082.2.patch, HIVE-12082.patch
>
>
> In mysql comparisons if any of the entries are null, then the result is null.
> [https://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html|https://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html]
> and
> [https://dev.mysql.com/doc/refman/5.0/en/type-conversion.html|https://dev.mysql.com/doc/refman/5.0/en/type-conversion.html].
> This can be demonstrated by the following mysql query:
> {noformat}
> mysql> select greatest(1, null) from test;
> +---+
> | greatest(1, null) |
> +---+
> | NULL |
> +---+
> 1 row in set (0.00 sec)
> mysql> select greatest(-1, null) from test;
> ++
> | greatest(-1, null) |
> ++
> | NULL |
> ++
> 1 row in set (0.00 sec)
> {noformat}
> This is in contrast to Hive, where null are ignored in the comparisons.
> {noformat}
> hive> select greatest(null, 1) from test;
> OK
> 1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13084) Vectorization add support for PROJECTION Multi-AND/OR
[
https://issues.apache.org/jira/browse/HIVE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366769#comment-15366769
]
Lefty Leverenz commented on HIVE-13084:
---
Reverted by HIVE-14182.
> Vectorization add support for PROJECTION Multi-AND/OR
> -
>
> Key: HIVE-13084
> URL: https://issues.apache.org/jira/browse/HIVE-13084
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Rajesh Balamohan
>Assignee: Matt McCline
> Fix For: 2.1.0
>
> Attachments: HIVE-13084.01.patch, HIVE-13084.02.patch,
> HIVE-13084.03.patch, HIVE-13084.04.patch, HIVE-13084.05.patch,
> HIVE-13084.06.patch, HIVE-13084.07.patch, vector_between_date.q
>
>
> When there is case statement in group by, hive throws unable to vectorize
> exception.
> e.g query just to demonstrate the problem
> {noformat}
> explain select l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END as wk from lineitem_test_l_shipdate_ts
> group by l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END;
> org.apache.hadoop.hive.ql.metadata.HiveException: Could not vectorize
> expression: org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2
> File Output Operator [FS_7]
> Group By Operator [GBY_5] (rows=888777234 width=108)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=1777554469 width=108)
> Output:["_col0","_col1"],keys:_col0, _col1
> Select Operator [SEL_1] (rows=1777554469 width=108)
> Output:["_col0","_col1"]
> TableScan [TS_0] (rows=1777554469 width=108)
>
> rajesh@lineitem_test_l_shipdate_ts,lineitem_test_l_shipdate_ts,Tbl:COMPLETE,Col:NONE,Output:["l_partkey","l_commitdate"]
> {noformat}
> \cc [~mmccline], [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13924) (Vectorization) Error evaluating ((bool0 and (not bool1)) or (bool1 and (not bool0)))
[
https://issues.apache.org/jira/browse/HIVE-13924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366768#comment-15366768
]
Lefty Leverenz commented on HIVE-13924:
---
Reverted by HIVE-14182.
> (Vectorization) Error evaluating ((bool0 and (not bool1)) or (bool1 and (not
> bool0)))
> -
>
> Key: HIVE-13924
> URL: https://issues.apache.org/jira/browse/HIVE-13924
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0, 2.2.0
>
> Attachments: HIVE-13924.01.patch
>
>
> Scratch column(s) in child expressions shouldn't be returned to the pool for
> PROJECTION.
> Problem introduced with HIVE-13084.
> Symptom:
> {code}
> Caused by: java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.ColOrCol.evaluate(ColOrCol.java:544)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14034) Vectorization may fail with compex OR conditions
[
https://issues.apache.org/jira/browse/HIVE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366766#comment-15366766
]
Lefty Leverenz commented on HIVE-14034:
---
Reverted by HIVE-14182.
> Vectorization may fail with compex OR conditions
>
>
> Key: HIVE-14034
> URL: https://issues.apache.org/jira/browse/HIVE-14034
> Project: Hive
> Issue Type: Bug
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14034.patch
>
>
> Example callstack for reference, should somebody hit this:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 288
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.StringGroupColEqualStringGroupScalarBase.evaluate(StringGroupColEqualStringGroupScalarBase.java:74)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.ColOrCol.evaluate(ColOrCol.java:268)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:121)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringScalarStringGroupColumn.evaluate(IfExprStringScalarStringGroupColumn.java:56)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:132)
> ... 23 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366749#comment-15366749 ] Lefty Leverenz commented on HIVE-14182: --- Thanks Matt. > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
[ https://issues.apache.org/jira/browse/HIVE-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366713#comment-15366713 ] Mohit Sabharwal commented on HIVE-14187: Thanks [~vgumashta]. I took a quick look at HIVE-7353 and had a question. Regarding the comment "the threadpools keep a certain number of threads live and kill excess threads after a configurable keepAliveTime expires ... ideally this is where we'd plug in code to close the JDOPersistanceManager" -- this applies only to those threads where JDOPersistanceManager#close was not already called, correct ? In the remote metastore case, deleteContext() will get executed when the underling transport gets closed for that thread (in WorkerProcess Runnable run() method). https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/server/TThreadPoolServer.java#L300 In this patch, we call cleanupRawStore inside deleteContext() (which calls JDOPersistanceManager#close) (If the client explicitly already called IMetaStoreClient#close on that connection, then the deleteContext() triggered cleanup will be a no-op.) I haven't looked at ThreadPoolExecutor code, but docs say "even core threads are initially created and started only when new tasks arrive", i.e. threads won't get pre-created (only to be killed later after keepalive timeout), but will get created when request arrives (which means there will be a transport associated with it, which will trigger deleteContext() upon connection break). IOW, thread goes back to the pool only after the deleteContext is called. And subsequently may get killed after keepalive timeout. So, unless someone manually kills the thread, it seems to me that deleteContext() covers the cleanup. (deleteContext() call is in the finally block so that covers errors/exceptions). What do you think ? > JDOPersistenceManager objects remain cached if MetaStoreClient#close is not > called > -- > > Key: HIVE-14187 > URL: https://issues.apache.org/jira/browse/HIVE-14187 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-14187.patch > > > JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by > DataNuclues. > A new JDOPersistenceManager object gets created for every HMS thread since > ObjectStore is a thread local. > In non-embedded metastore mode, JDOPersistenceManager associated with a > thread only gets cleaned up if IMetaStoreClient#close is called by the client > (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close > which in turn removes the object from cache in > JDOPersistenceManagerFactory#releasePersistenceManager > https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), > i.e. the object will remain cached if client does not call close. > For example: If one interrupts out of hive CLI shell (instead of using > 'exit;' command), SessionState#close does not get called, and hence > IMetaStoreClient#close does not get called. > Instead of relying the client to call close, it's cleaner to automatically > perform RawStore related cleanup at the server end via deleteContext() which > gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Resolution: Fixed
Status: Resolved (was: Patch Available)
committed to master. I'll file a separate JIRA to backport, since the test run
would need to be done given the scope of the changes
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.04.patch,
> HIVE-13945.05.patch, HIVE-13945.06.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1538#comment-1538
]
Sergey Shelukhin commented on HIVE-13945:
-
Thanks for the review!
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.04.patch,
> HIVE-13945.05.patch, HIVE-13945.06.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Fix Version/s: 2.2.0
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.04.patch,
> HIVE-13945.05.patch, HIVE-13945.06.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Target Version/s: 2.2.0 (was: 1.3.0, 2.2.0)
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.04.patch,
> HIVE-13945.05.patch, HIVE-13945.06.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366652#comment-15366652 ] Matt McCline commented on HIVE-14182: - Updated errata.txt for this issue. > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14188) LLAPIF: wrong user field is used from the token
[ https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366617#comment-15366617 ] Jason Dere commented on HIVE-14188: --- +1 > LLAPIF: wrong user field is used from the token > --- > > Key: HIVE-14188 > URL: https://issues.apache.org/jira/browse/HIVE-14188 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14188.patch > > > realUser is not usually set in all cases for delegation tokens; we should use > the owner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12915) Tez session pool has concurrency issues during init
[ https://issues.apache.org/jira/browse/HIVE-12915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366590#comment-15366590 ] Sergey Shelukhin commented on HIVE-12915: - Don't recall. Maybe? Backports welcome ;) > Tez session pool has concurrency issues during init > --- > > Key: HIVE-12915 > URL: https://issues.apache.org/jira/browse/HIVE-12915 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-12915.01.patch, HIVE-12915.02.patch, > HIVE-12915.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14188) LLAPIF: wrong user field is used from the token
[ https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14188: Attachment: HIVE-14188.patch Tiny patch. [~jdere] can you take a look? > LLAPIF: wrong user field is used from the token > --- > > Key: HIVE-14188 > URL: https://issues.apache.org/jira/browse/HIVE-14188 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14188.patch > > > realUser is not usually set in all cases for delegation tokens; we should use > the owner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14188) LLAPIF: wrong user field is used from the token
[ https://issues.apache.org/jira/browse/HIVE-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14188: Status: Patch Available (was: Open) > LLAPIF: wrong user field is used from the token > --- > > Key: HIVE-14188 > URL: https://issues.apache.org/jira/browse/HIVE-14188 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14188.patch > > > realUser is not usually set in all cases for delegation tokens; we should use > the owner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12915) Tez session pool has concurrency issues during init
[ https://issues.apache.org/jira/browse/HIVE-12915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366578#comment-15366578 ] Vaibhav Gumashta commented on HIVE-12915: - [~sershe] Should this go into branch-1 as well? > Tez session pool has concurrency issues during init > --- > > Key: HIVE-12915 > URL: https://issues.apache.org/jira/browse/HIVE-12915 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Fix For: 2.0.0 > > Attachments: HIVE-12915.01.patch, HIVE-12915.02.patch, > HIVE-12915.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
[ https://issues.apache.org/jira/browse/HIVE-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366575#comment-15366575 ] Vaibhav Gumashta commented on HIVE-14187: - [~mohitsabharwal] Thanks for the patch. However, this will still not completely fix the issue as the the JDOPersistenceManager is cached in thread locals and threads in a threadpool can die randomly causing the objects to get accumulated. This was fixed in HIVE-7353, HIVE-9831 for HS2. Maybe we can take a similar approach here? > JDOPersistenceManager objects remain cached if MetaStoreClient#close is not > called > -- > > Key: HIVE-14187 > URL: https://issues.apache.org/jira/browse/HIVE-14187 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-14187.patch > > > JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by > DataNuclues. > A new JDOPersistenceManager object gets created for every HMS thread since > ObjectStore is a thread local. > In non-embedded metastore mode, JDOPersistenceManager associated with a > thread only gets cleaned up if IMetaStoreClient#close is called by the client > (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close > which in turn removes the object from cache in > JDOPersistenceManagerFactory#releasePersistenceManager > https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), > i.e. the object will remain cached if client does not call close. > For example: If one interrupts out of hive CLI shell (instead of using > 'exit;' command), SessionState#close does not get called, and hence > IMetaStoreClient#close does not get called. > Instead of relying the client to call close, it's cleaner to automatically > perform RawStore related cleanup at the server end via deleteContext() which > gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
[ https://issues.apache.org/jira/browse/HIVE-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14187: --- Status: Patch Available (was: Open) > JDOPersistenceManager objects remain cached if MetaStoreClient#close is not > called > -- > > Key: HIVE-14187 > URL: https://issues.apache.org/jira/browse/HIVE-14187 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-14187.patch > > > JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by > DataNuclues. > A new JDOPersistenceManager object gets created for every HMS thread > local since ObjectStore is a thread local. > In non-embedded metastore mode, JDOPersistenceManager associated with a > thread only gets cleaned up if IMetaStoreClient#close is called by the client > (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close > which in turn removes the object from cache in > JDOPersistenceManagerFactory#releasePersistenceManager > https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), > i.e. the object will remain cached if client does not call close. > For example: If one interrupts out of hive CLI shell (instead of using > 'exit;' command), SessionState#close does not get called, and hence > IMetaStoreClient#close does not get called. > Instead of relying the client to call close, it's cleaner to automatically > perform RawStore related cleanup at the server end via deleteContext() which > gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14187) JDOPersistenceManager objects remain cached if MetaStoreClient#close is not called
[ https://issues.apache.org/jira/browse/HIVE-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14187: --- Attachment: HIVE-14187.patch > JDOPersistenceManager objects remain cached if MetaStoreClient#close is not > called > -- > > Key: HIVE-14187 > URL: https://issues.apache.org/jira/browse/HIVE-14187 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-14187.patch > > > JDOPersistenceManager objects are cached in JDOPersistenceManagerFactory by > DataNuclues. > A new JDOPersistenceManager object gets created for every HMS thread > local since ObjectStore is a thread local. > In non-embedded metastore mode, JDOPersistenceManager associated with a > thread only gets cleaned up if IMetaStoreClient#close is called by the client > (which calls ObjectStore#shutdown which calls JDOPersistenceManager#close > which in turn removes the object from cache in > JDOPersistenceManagerFactory#releasePersistenceManager > https://github.com/datanucleus/datanucleus-api-jdo/blob/master/src/main/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L1271), > i.e. the object will remain cached if client does not call close. > For example: If one interrupts out of hive CLI shell (instead of using > 'exit;' command), SessionState#close does not get called, and hence > IMetaStoreClient#close does not get called. > Instead of relying the client to call close, it's cleaner to automatically > perform RawStore related cleanup at the server end via deleteContext() which > gets called when the server detects a lost/closed connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366550#comment-15366550 ] Siddharth Seth commented on HIVE-14111: --- Lets wait for HiveQA - it's identified a bunch of issues already. That said, it seems to be identifying issues with QTestUtil rather than the main code. Hopefully some tests are covering the main flow. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.patch, > sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14135) beeline output not formatted correctly for large column widths
[ https://issues.apache.org/jira/browse/HIVE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366528#comment-15366528 ] Vihang Karajgaonkar commented on HIVE-14135: Thanks [~thejas] for your suggestions. I will add them to the patch and submit it again. > beeline output not formatted correctly for large column widths > -- > > Key: HIVE-14135 > URL: https://issues.apache.org/jira/browse/HIVE-14135 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.2.0 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-14135.1.patch, longKeyValues.txt, output_after.txt, > output_before.txt > > > If the column width is too large then beeline uses the maximum column width > when normalizing all the column widths. In order to reproduce the issue, run > set -v; > Once the configuration variables is classpath which can be extremely large > width (41k characters in my environment). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive generates inefficient metastore queries for TPCDS tables with 1800+ partitions leading to higher compile time
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366474#comment-15366474
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-13995:
--
[~ashutoshc] There is a NPE issue with the previous patch which covers a bunch
of the failures. I am looking at the remaining ones. Will hopefully post a
patch by later today
> Hive generates inefficient metastore queries for TPCDS tables with 1800+
> partitions leading to higher compile time
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these columns are required for hive query optimization.
--
This message was
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366450#comment-15366450
]
Ashutosh Chauhan commented on HIVE-13945:
-
+1
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.04.patch,
> HIVE-13945.05.patch, HIVE-13945.06.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14158) deal with derived column names
[
https://issues.apache.org/jira/browse/HIVE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366442#comment-15366442
]
Pengcheng Xiong commented on HIVE-14158:
more info
{code}
SQL> create view v1 as select a from t1;
View created.
SQL> select * from v1;
A
--
10
10
{code}
> deal with derived column names
> --
>
> Key: HIVE-14158
> URL: https://issues.apache.org/jira/browse/HIVE-14158
> Project: Hive
> Issue Type: Sub-task
> Components: Security
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.1.0
>
> Attachments: HIVE-14158.01.patch, HIVE-14158.02.patch,
> HIVE-14158.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Attachment: HIVE-14035.09.patch Patch (#09) same as the previous patch (#08) with no code modifications, except that it is rebased with master and has trailing whitespaces fixed. Mirrors the patch uploaded at review board. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: Open) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Open (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366408#comment-15366408 ] Saket Saurabh commented on HIVE-14035: -- Link for review board for this patch: https://reviews.apache.org/r/49766/ Thanks > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14124) Spark app name should be in line with MapReduce app name when using Hive On Spark
[ https://issues.apache.org/jira/browse/HIVE-14124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366367#comment-15366367 ] Xuefu Zhang commented on HIVE-14124: Well, this is by design. Hive on Spark reuses the AM for the whole user session, so the app name isn't mapped to a single user query. > Spark app name should be in line with MapReduce app name when using Hive On > Spark > - > > Key: HIVE-14124 > URL: https://issues.apache.org/jira/browse/HIVE-14124 > Project: Hive > Issue Type: Improvement > Components: Spark >Affects Versions: 1.1.0 >Reporter: Thomas Scott >Priority: Minor > > When using the spark execution engine the jobs submitted to YARN are > submitted with name "Hive On Spark" whereas in mr execution engine the name > contains the query executed. This is overrideable via spark.app.name but it > should automatically fill out the query executed in line with the mr engine. > Example: > set hive.execution.engine=spark; > Select count(*) from sometable; > > -> Launched YARN Job description: Hive On Spark > set hive.execution.engine=mr; > Select count(*) from sometable; > > -> Launched YARN Job description: Select count(*) from sometable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13882) When hive.server2.async.exec.async.compile is turned on, from JDBC we will get "The query did not generate a result set"
[
https://issues.apache.org/jira/browse/HIVE-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366272#comment-15366272
]
Aihua Xu commented on HIVE-13882:
-
[~vgumashta] Yes. Please commit to branch-2.1 as well. Since
hive.server2.async.exec.async.compile is default to false, so I didn't put to
2.1, but please do. Thanks.
> When hive.server2.async.exec.async.compile is turned on, from JDBC we will
> get "The query did not generate a result set"
> -
>
> Key: HIVE-13882
> URL: https://issues.apache.org/jira/browse/HIVE-13882
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13882.1.patch, HIVE-13882.2.patch
>
>
> The following would fail with "The query did not generate a result set"
> stmt.execute("SET hive.driver.parallel.compilation=true");
> stmt.execute("SET hive.server2.async.exec.async.compile=true");
> ResultSet res = stmt.executeQuery("SELECT * FROM " + tableName);
> res.next();
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366262#comment-15366262
]
Sahil Takiar commented on HIVE-13278:
-
Yes, makes sense. The current solution should work for most cases.
Thanks for all the help!
> Many redundant 'File not found' messages appeared in container log during
> query execution with Hive on Spark
>
>
> Key: HIVE-13278
> URL: https://issues.apache.org/jira/browse/HIVE-13278
> Project: Hive
> Issue Type: Bug
> Environment: Hive on Spark engine
> Found based on :
> Apache Hive 2.0.0
> Apache Spark 1.6.0
>Reporter: Xin Hao
>Assignee: Sahil Takiar
>Priority: Minor
>
> Many redundant 'File not found' messages appeared in container log during
> query execution with Hive on Spark.
> Certainly, it doesn't prevent the query from running successfully. So mark it
> as Minor currently.
> Error message example:
> {noformat}
> 16/03/14 01:45:06 INFO exec.Utilities: File not found: File does not exist:
> /tmp/hive/hadoop/2d378538-f5d3-493c-9276-c62dd6634fb4/hive_2016-03-14_01-44-16_835_623058724409492515-6/-mr-10010/0a6d0cae-1eb3-448c-883b-590b3b198a73/reduce.xml
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:565)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14146) Column comments with "\n" character "corrupts" table metadata
[
https://issues.apache.org/jira/browse/HIVE-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366154#comment-15366154
]
Aihua Xu commented on HIVE-14146:
-
[~pvary] I notice that we are calling
HiveStringUtils.esapeJava(escapeHiveCommand(string)). Does it make sense to
move HiveStringUtils.esapeJava() call inside escapeHiveCommand() so it will
control escaping characters in one place?
> Column comments with "\n" character "corrupts" table metadata
> -
>
> Key: HIVE-14146
> URL: https://issues.apache.org/jira/browse/HIVE-14146
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14146.2.patch, HIVE-14146.3.patch, HIVE-14146.patch
>
>
> Create a table with the following(noting the \n in the COMMENT):
> {noformat}
> CREATE TABLE commtest(first_nm string COMMENT 'Indicates First name\nof an
> individual’);
> {noformat}
> Describe shows that now the metadata is messed up:
> {noformat}
> beeline> describe commtest;
> +---++---+--+
> | col_name | data_type |comment|
> +---++---+--+
> | first_nm | string | Indicates First name |
> | of an individual | NULL | NULL |
> +---++---+--+
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13604) Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
[
https://issues.apache.org/jira/browse/HIVE-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-13604:
Status: Patch Available (was: Open)
'IF NOT EXISTS' specified it should skip from analysis phase only if
view/database exists, no need to go to the execution.
> Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
> ---
>
> Key: HIVE-13604
> URL: https://issues.apache.org/jira/browse/HIVE-13604
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Yuriy Plysyuk
>Assignee: Chinna Rao Lalam
>Priority: Trivial
> Attachments: HIVE-13604.patch
>
>
> When trying to create view that exists with statement:
> CREATE VIEW IF NOT EXISTS dummy_table ...
> Next error is logged:
> ERROR RetryingHMSHandler:190 - AlreadyExistsException(message:Table
> dummy_view already exists)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1296)
> ...
> The same when creating schema using:
> CREATE SCHEMA IF NOT EXISTS ...
> Error should not be logged as it confuses.
> For
> CREATE TABLE IF NOT EXISTS ...
> it works fine. I checked that there is code to handle this in:
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> could you please add similar check for creating views and schema?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13604) Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
[
https://issues.apache.org/jira/browse/HIVE-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-13604:
Attachment: HIVE-13604.patch
> Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
> ---
>
> Key: HIVE-13604
> URL: https://issues.apache.org/jira/browse/HIVE-13604
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Yuriy Plysyuk
>Assignee: Chinna Rao Lalam
>Priority: Trivial
> Attachments: HIVE-13604.patch
>
>
> When trying to create view that exists with statement:
> CREATE VIEW IF NOT EXISTS dummy_table ...
> Next error is logged:
> ERROR RetryingHMSHandler:190 - AlreadyExistsException(message:Table
> dummy_view already exists)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1296)
> ...
> The same when creating schema using:
> CREATE SCHEMA IF NOT EXISTS ...
> Error should not be logged as it confuses.
> For
> CREATE TABLE IF NOT EXISTS ...
> it works fine. I checked that there is code to handle this in:
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> could you please add similar check for creating views and schema?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13604) Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
[
https://issues.apache.org/jira/browse/HIVE-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam reassigned HIVE-13604:
---
Assignee: Chinna Rao Lalam
> Do not log AlreadyExistsException when "IF NOT EXISTS" is used.
> ---
>
> Key: HIVE-13604
> URL: https://issues.apache.org/jira/browse/HIVE-13604
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.0.0
>Reporter: Yuriy Plysyuk
>Assignee: Chinna Rao Lalam
>Priority: Trivial
>
> When trying to create view that exists with statement:
> CREATE VIEW IF NOT EXISTS dummy_table ...
> Next error is logged:
> ERROR RetryingHMSHandler:190 - AlreadyExistsException(message:Table
> dummy_view already exists)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1296)
> ...
> The same when creating schema using:
> CREATE SCHEMA IF NOT EXISTS ...
> Error should not be logged as it confuses.
> For
> CREATE TABLE IF NOT EXISTS ...
> it works fine. I checked that there is code to handle this in:
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable
> // check for existence of table
> if (ifNotExists) {
> try {
> Table table = getTable(qualifiedTabName, false);
> if (table != null) { // table exists
> return null;
> }
> could you please add similar check for creating views and schema?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14184) Adding test for limit pushdown in presence of grouping sets
[ https://issues.apache.org/jira/browse/HIVE-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14184 started by Jesus Camacho Rodriguez. -- > Adding test for limit pushdown in presence of grouping sets > --- > > Key: HIVE-14184 > URL: https://issues.apache.org/jira/browse/HIVE-14184 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14184.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14184) Adding test for limit pushdown in presence of grouping sets
[ https://issues.apache.org/jira/browse/HIVE-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14184: --- Attachment: HIVE-14184.patch > Adding test for limit pushdown in presence of grouping sets > --- > > Key: HIVE-14184 > URL: https://issues.apache.org/jira/browse/HIVE-14184 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14184.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14184) Adding test for limit pushdown in presence of grouping sets
[ https://issues.apache.org/jira/browse/HIVE-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14184: --- Status: Patch Available (was: In Progress) > Adding test for limit pushdown in presence of grouping sets > --- > > Key: HIVE-14184 > URL: https://issues.apache.org/jira/browse/HIVE-14184 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14184.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14156) Problem with Chinese characters as partition value when using MySQL
[ https://issues.apache.org/jira/browse/HIVE-14156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365954#comment-15365954 ] Rui Li commented on HIVE-14156: --- According to wiki, the minimum supported MySQL version is 5.6.17. > Problem with Chinese characters as partition value when using MySQL > --- > > Key: HIVE-14156 > URL: https://issues.apache.org/jira/browse/HIVE-14156 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 2.0.0 >Reporter: Bing Li >Assignee: Bing Li > > Steps to reproduce: > create table t1 (name string, age int) partitioned by (city string) row > format delimited fields terminated by ','; > load data local inpath '/tmp/chn-partition.txt' overwrite into table t1 > partition (city='北京'); > The content of /tmp/chn-partition.txt: > 小明,20 > 小红,15 > 张三,36 > 李四,50 > When check the partition value in MySQL, it shows ?? instead of "北京". > When run "drop table t1", it will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365940#comment-15365940 ] Peter Vary commented on HIVE-14100: --- Review request sent: https://reviews.apache.org/r/49748/ > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14100.2.patch, HIVE-14100.patch > > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateAuthenticator the current_user returns the username which > is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365931#comment-15365931 ] Peter Vary commented on HIVE-14100: --- The errors are not related. > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14100.2.patch, HIVE-14100.patch > > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateAuthenticator the current_user returns the username which > is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365756#comment-15365756 ] Lefty Leverenz edited comment on HIVE-14182 at 7/7/16 8:43 AM: --- [~mmccline], the three commits don't give this JIRA number (HIVE-14182) so would you please update the errata.txt file? Here are the commit numbers: * 5c58dceeaf662b6314eedb9afa01a2896657ef77 -- Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)" * 22df7a8441ca85ad7f64e5191d4675f2f36a0664 -- Revert "HIVE-14034 : Vectorization may fail with compex OR conditions (Sergey Shelukhin, reviewed by Matt McCline) * 223350894fe5aa653668e9f39e43218e514f2b24 -- Revert "HIVE-13924: (Vectorization) Error evaluating ((bool0 and (not bool1)) or (bool1 and (not bool0))) (Matt McCline, reviewed by Gopal Vijayaraghavan)" See HIVE-11876 for an example of updating errata.txt. was (Author: le...@hortonworks.com): [~mmccline], the commit doesn't give the JIRA number (HIVE-14182) so would you please update the errata.txt file? It's commit # 5c58dceeaf662b6314eedb9afa01a2896657ef77. See HIVE-11876 for an example. > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14163) LLAP: use different kerberized/unkerberized zk paths for registry
[ https://issues.apache.org/jira/browse/HIVE-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365769#comment-15365769 ] Lefty Leverenz commented on HIVE-14163: --- Doc note: This adds *hive.llap.zk.registry.namespace* to HiveConf.java, so it will need to be documented in the LLAP section of Configuration Properties for 2.1.1 and 2.2.0. * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added TODOC2.1.1 and TODOC2.2.0 labels. > LLAP: use different kerberized/unkerberized zk paths for registry > - > > Key: HIVE-14163 > URL: https://issues.apache.org/jira/browse/HIVE-14163 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14163.01.patch, HIVE-14163.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14163) LLAP: use different kerberized/unkerberized zk paths for registry
[ https://issues.apache.org/jira/browse/HIVE-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14163: -- Labels: TODOC2.1.1 TODOC2.2 (was: ) > LLAP: use different kerberized/unkerberized zk paths for registry > - > > Key: HIVE-14163 > URL: https://issues.apache.org/jira/browse/HIVE-14163 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1.1, TODOC2.2 > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14163.01.patch, HIVE-14163.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365756#comment-15365756 ] Lefty Leverenz commented on HIVE-14182: --- [~mmccline], the commit doesn't give the JIRA number (HIVE-14182) so would you please update the errata.txt file? It's commit # 5c58dceeaf662b6314eedb9afa01a2896657ef77. See HIVE-11876 for an example. > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365723#comment-15365723
]
Lefty Leverenz commented on HIVE-13884:
---
You need a Confluence account:
* [About This Wiki -- How to get permission to edit |
https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit]
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14182: Fix Version/s: 2.2.0 > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.2.0 > > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[
https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365710#comment-15365710
]
Matt McCline edited comment on HIVE-14182 at 7/7/16 7:06 AM:
-
Committed to master the revert of the original HIVE-13084 and 2 bug fixes.
{code}
remote: hive git commit: Revert "HIVE-13084: Vectorization add support for
PROJECTION Multi-AND/OR (Matt McCline, reviewed by Sergey Shelukhin)"
remote: hive git commit: Revert "HIVE-13924: (Vectorization) Error evaluating
((bool0 and (not bool1)) or (bool1 and (not bool0))) (Matt McCline, reviewed by
Gopal Vijayaraghavan)"
remote: hive git commit: Revert "HIVE-14034 : Vectorization may fail with
compex OR conditions (Sergey Shelukhin, reviewed by Matt McCline)
{code}
was (Author: mmccline):
Committed to master.
> Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
> -
>
> Key: HIVE-14182
> URL: https://issues.apache.org/jira/browse/HIVE-14182
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13084.revert.patch
>
>
> To many issues with scratch column allocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365710#comment-15365710 ] Matt McCline commented on HIVE-14182: - Committed to master. > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14159) sorting of tuple array using multiple field[s]
[
https://issues.apache.org/jira/browse/HIVE-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365699#comment-15365699
]
Carl Steinbach commented on HIVE-14159:
---
Hi [~simanchal], I left some comments on RB. Also, it looks like there is a
test failure in show_functions. Please take a look.
> sorting of tuple array using multiple field[s]
> --
>
> Key: HIVE-14159
> URL: https://issues.apache.org/jira/browse/HIVE-14159
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Simanchal Das
>Assignee: Simanchal Das
> Labels: patch
> Attachments: HIVE-14159.1.patch, HIVE-14159.2.patch
>
>
> Problem Statement:
> When we are working with complex structure of data like avro.
> Most of the times we are encountering array contains multiple tuples and each
> tuple have struct schema.
> Suppose here struct schema is like below:
> {noformat}
> {
> "name": "employee",
> "type": [{
> "type": "record",
> "name": "Employee",
> "namespace": "com.company.Employee",
> "fields": [{
> "name": "empId",
> "type": "int"
> }, {
> "name": "empName",
> "type": "string"
> }, {
> "name": "age",
> "type": "int"
> }, {
> "name": "salary",
> "type": "double"
> }]
> }]
> }
> {noformat}
> Then while running our hive query complex array looks like array of employee
> objects.
> {noformat}
> Example:
> //(array>)
>
> Array[Employee(100,Foo,20,20990),Employee(500,Boo,30,50990),Employee(700,Harry,25,40990),Employee(100,Tom,35,70990)]
> {noformat}
> When we are implementing business use cases day to day life we are
> encountering problems like sorting a tuple array by specific field[s] like
> empId,name,salary,etc.
> Proposal:
> I have developed a udf 'sort_array_field' which will sort a tuple array by
> one or more fields in naural order.
> {noformat}
> Example:
> 1.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Salary");
> output:
> array[struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(500,Boo,30,50990),struct(100,Tom,35,70990)]
>
> 2.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,80990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Name","Salary");
> output:
> array[struct(500,Boo,30,50990),struct(500,Boo,30,80990),struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)]
> 3.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Name","Salary","Age);
> output:
> array[struct(500,Boo,30,50990),struct(500,Boo,30,80990),struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14159) sorting of tuple array using multiple field[s]
[
https://issues.apache.org/jira/browse/HIVE-14159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-14159:
--
Status: Open (was: Patch Available)
> sorting of tuple array using multiple field[s]
> --
>
> Key: HIVE-14159
> URL: https://issues.apache.org/jira/browse/HIVE-14159
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Simanchal Das
>Assignee: Simanchal Das
> Labels: patch
> Attachments: HIVE-14159.1.patch, HIVE-14159.2.patch
>
>
> Problem Statement:
> When we are working with complex structure of data like avro.
> Most of the times we are encountering array contains multiple tuples and each
> tuple have struct schema.
> Suppose here struct schema is like below:
> {noformat}
> {
> "name": "employee",
> "type": [{
> "type": "record",
> "name": "Employee",
> "namespace": "com.company.Employee",
> "fields": [{
> "name": "empId",
> "type": "int"
> }, {
> "name": "empName",
> "type": "string"
> }, {
> "name": "age",
> "type": "int"
> }, {
> "name": "salary",
> "type": "double"
> }]
> }]
> }
> {noformat}
> Then while running our hive query complex array looks like array of employee
> objects.
> {noformat}
> Example:
> //(array>)
>
> Array[Employee(100,Foo,20,20990),Employee(500,Boo,30,50990),Employee(700,Harry,25,40990),Employee(100,Tom,35,70990)]
> {noformat}
> When we are implementing business use cases day to day life we are
> encountering problems like sorting a tuple array by specific field[s] like
> empId,name,salary,etc.
> Proposal:
> I have developed a udf 'sort_array_field' which will sort a tuple array by
> one or more fields in naural order.
> {noformat}
> Example:
> 1.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Salary");
> output:
> array[struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(500,Boo,30,50990),struct(100,Tom,35,70990)]
>
> 2.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,80990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Name","Salary");
> output:
> array[struct(500,Boo,30,50990),struct(500,Boo,30,80990),struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)]
> 3.Select
> sort_array_field(array[struct(100,Foo,20,20990),struct(500,Boo,30,50990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)],"Name","Salary","Age);
> output:
> array[struct(500,Boo,30,50990),struct(500,Boo,30,80990),struct(100,Foo,20,20990),struct(700,Harry,25,40990),struct(100,Tom,35,70990)]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14182: Status: Patch Available (was: Open) > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-14182: --- Assignee: Matt McCline > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14182) Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR
[ https://issues.apache.org/jira/browse/HIVE-14182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14182: Attachment: HIVE-13084.revert.patch > Revert "HIVE-13084: Vectorization add support for PROJECTION Multi-AND/OR > - > > Key: HIVE-14182 > URL: https://issues.apache.org/jira/browse/HIVE-14182 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13084.revert.patch > > > To many issues with scratch column allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
