[jira] [Commented] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[ https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344758#comment-15344758 ] Abdullah Yousufi commented on HIVE-14074: - No I don't believe that should be an issue because the function registry is local to the session. So other sessions won't experience any changes with their functions until they reload their functions. And when the other sessions do reload their functions, they will be reading from the metastore, which is thread safe. > RELOAD FUNCTION should update dropped functions > --- > > Key: HIVE-14074 > URL: https://issues.apache.org/jira/browse/HIVE-14074 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Fix For: 2.2.0 > > Attachments: HIVE-14074.01.patch > > > Due to HIVE-2573, functions are stored in a per-session registry and only > loaded in from the metastore when hs2 or hive cli is started. Running RELOAD > FUNCTION in the current session is a way to force a reload of the functions, > so that changes that occurred in other running sessions will be reflected in > the current session, without having to restart the current session. However, > while functions that are created in other sessions will now appear in the > current session, functions that have been dropped are not removed from the > current session's registry. It seems inconsistent that created functions are > updated while dropped functions are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14062) Changes from HIVE-13502 overwritten by HIVE-13566
[ https://issues.apache.org/jira/browse/HIVE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14062: Fix Version/s: (was: 2.2.0) 2.1.0 > Changes from HIVE-13502 overwritten by HIVE-13566 > - > > Key: HIVE-14062 > URL: https://issues.apache.org/jira/browse/HIVE-14062 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.1.0 > > Attachments: HIVE-14062.1.patch > > > Appears that changes from HIVE-13566 overwrote the changes from HIVE-13502. I > will confirm with the author that it was inadvertent before I re-add it. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14062) Changes from HIVE-13502 overwritten by HIVE-13566
[ https://issues.apache.org/jira/browse/HIVE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14062: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.1 and master. We just have 2.1 released, so I guess it's 2.1.1 and 2.2.0. > Changes from HIVE-13502 overwritten by HIVE-13566 > - > > Key: HIVE-14062 > URL: https://issues.apache.org/jira/browse/HIVE-14062 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14062.1.patch > > > Appears that changes from HIVE-13566 overwrote the changes from HIVE-13502. I > will confirm with the author that it was inadvertent before I re-add it. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14062) Changes from HIVE-13502 overwritten by HIVE-13566
[ https://issues.apache.org/jira/browse/HIVE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344410#comment-15344410 ] Aihua Xu commented on HIVE-14062: - Thanks Naveen for the work to correct it. > Changes from HIVE-13502 overwritten by HIVE-13566 > - > > Key: HIVE-14062 > URL: https://issues.apache.org/jira/browse/HIVE-14062 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14062.1.patch > > > Appears that changes from HIVE-13566 overwrote the changes from HIVE-13502. I > will confirm with the author that it was inadvertent before I re-add it. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both
[
https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344412#comment-15344412
]
Sergio Peña commented on HIVE-14001:
Hey [~pvary], I applied the patch, and when I run {{TestBeelineArgParsing}}
test, it failed.
{noformat}
---
T E S T S
---
Running org.apache.hive.beeline.TestBeelineArgParsing
Tests run: 28, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 2.936 sec <<<
FAILURE! - in org.apache.hive.beeline.TestBeelineArgParsing
testAddLocalJar[0](org.apache.hive.beeline.TestBeelineArgParsing) Time
elapsed: 0.114 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hive.beeline.Commands.addlocaldrivername(Commands.java:149)
at
org.apache.hive.beeline.TestBeelineArgParsing$TestBeeline.addlocaldrivername(TestBeelineArgParsing.java:86)
at
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar(TestBeelineArgParsing.java:242)
testAddLocalJarWithoutAddDriverClazz[0](org.apache.hive.beeline.TestBeelineArgParsing)
Time elapsed: 0.051 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz(TestBeelineArgParsing.java:256)
testAddLocalJar[1](org.apache.hive.beeline.TestBeelineArgParsing) Time
elapsed: 0.028 sec <<< ERROR!
java.lang.NullPointerException: null
at
org.apache.hive.beeline.Commands.addlocaldrivername(Commands.java:149)
at
org.apache.hive.beeline.TestBeelineArgParsing$TestBeeline.addlocaldrivername(TestBeelineArgParsing.java:86)
at
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar(TestBeelineArgParsing.java:242)
Results :
Tests in error:
TestBeelineArgParsing.testAddLocalJar:242 » NullPointer
TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz:256 NullPointer
TestBeelineArgParsing.testAddLocalJar:242 » NullPointer
Tests run: 28, Failures: 0, Errors: 3, Skipped: 0
{noformat}
> beeline doesn't give out an error when takes either "-e" or "-f" in command
> instead of both
> ---
>
> Key: HIVE-14001
> URL: https://issues.apache.org/jira/browse/HIVE-14001
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 0.10.0, 2.0.1
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Trivial
> Attachments: HIVE-14001.2.patch, HIVE-14001.patch
>
>
> When providing both arguments there should be an error message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Status: Patch Available (was: In Progress)
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344462#comment-15344462
]
Jesus Camacho Rodriguez commented on HIVE-13982:
Updated three remaining q files.
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Status: Open (was: Patch Available)
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344508#comment-15344508
]
Ashutosh Chauhan commented on HIVE-14070:
-
TestOperationLoggingAPIWithMr failures look related. Probably, test code needs
update.
[~sershe] Can you also review this perflogger change?
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344531#comment-15344531
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812327/HIVE-14035.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10253 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/220/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/220/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812327 - PreCommit-HIVE-MASTER-Build
> Enable predicate pushdown to delta files created by ACID Transactions
> -
>
> Key: HIVE-14035
> URL: https://issues.apache.org/jira/browse/HIVE-14035
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Saket Saurabh
>Assignee: Saket Saurabh
> Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch,
> HIVE-14035.patch
>
>
> In current Hive version, delta files created by ACID transactions do not
> allow predicate pushdown if they contain any update/delete events. This is
> done to preserve correctness when following a multi-version approach during
> event collapsing, where an update event overwrites an existing insert event.
> This JIRA proposes to split an update event into a combination of a delete
> event followed by a new insert event, that can enable predicate push down to
> all delta files without breaking correctness. To support backward
> compatibility for this feature, this JIRA also proposes to add some sort of
> versioning to ACID that can allow different versions of ACID transactions to
> co-exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344536#comment-15344536
]
Ferdinand Xu commented on HIVE-14037:
-
Thanks [~aihuaxu] reaching me about this. It may be an issue in other non HS2
node since newly added jars are not distributed to these nodes. I am not so
sure whether there is a better solution than tmpjars of JobConf. Maybe we could
have a try in this way.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7443) Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs
[ https://issues.apache.org/jira/browse/HIVE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-7443: --- Attachment: HIVE-7443.3.patch Attached patch-3: missing the return which causes the test failures. > Fix HiveConnection to communicate with Kerberized Hive JDBC server and > alternative JDKs > --- > > Key: HIVE-7443 > URL: https://issues.apache.org/jira/browse/HIVE-7443 > Project: Hive > Issue Type: Bug > Components: JDBC, Security >Affects Versions: 0.12.0, 0.13.1 > Environment: Kerberos > Run Hive server2 and client with IBM JDK7.1 >Reporter: Yu Gao >Assignee: Aihua Xu > Attachments: HIVE-7443.2.patch, HIVE-7443.3.patch, HIVE-7443.patch > > > Hive Kerberos authentication has been enabled in my cluster. I ran kinit to > initialize the current login user's ticket cache successfully, and then tried > to use beeline to connect to Hive Server2, but failed. After I manually added > some logging to catch the failure exception, this is what I got that caused > the failure: > beeline> !connect > jdbc:hive2://:1/default;principal=hive/@REALM.COM > org.apache.hive.jdbc.HiveDriver > scan complete in 2ms > Connecting to > jdbc:hive2://:1/default;principal=hive/@REALM.COM > Enter password for > jdbc:hive2://:1/default;principal=hive/@REALM.COM: > 14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport > javax.security.sasl.SaslException: Failed to open client transport [Caused by > java.io.IOException: Could not instantiate SASL transport] > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78) > at > org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342) > at > org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200) > at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178) > at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:198) > at > org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) > at > org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186) > at org.apache.hive.beeline.Commands.connect(Commands.java:959) > at org.apache.hive.beeline.Commands.connect(Commands.java:880) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at > org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) > at java.lang.reflect.Method.invoke(Method.java:619) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.io.IOException: Could not instantiate SASL transport > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177) > at > org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74) > ... 24 more > Caused by: javax.security.sasl.SaslException: Failure to initialize security > context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0 > major string: Invalid credentials > minor string: SubjectCredFinder: no JAAS Subject] > at > com.ibm.security.sasl.gsskerb.GssKrb5Client.(GssKrb5Client.java:131) > at > com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53) > at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362) > at > org.apache.thrift.transport.TSaslClientTransport.(TSaslClientTransport.java:72) > at > org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:169) > ... 25 more > Caused by:
[jira] [Updated] (HIVE-14062) Changes from HIVE-13502 overwritten by HIVE-13566
[ https://issues.apache.org/jira/browse/HIVE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14062: Fix Version/s: 2.2.0 > Changes from HIVE-13502 overwritten by HIVE-13566 > - > > Key: HIVE-14062 > URL: https://issues.apache.org/jira/browse/HIVE-14062 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0 > > Attachments: HIVE-14062.1.patch > > > Appears that changes from HIVE-13566 overwrote the changes from HIVE-13502. I > will confirm with the author that it was inadvertent before I re-add it. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14062) Changes from HIVE-13502 overwritten by HIVE-13566
[ https://issues.apache.org/jira/browse/HIVE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14062: Fix Version/s: (was: 2.1.0) 2.1.1 2.2.0 > Changes from HIVE-13502 overwritten by HIVE-13566 > - > > Key: HIVE-14062 > URL: https://issues.apache.org/jira/browse/HIVE-14062 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14062.1.patch > > > Appears that changes from HIVE-13566 overwrote the changes from HIVE-13502. I > will confirm with the author that it was inadvertent before I re-add it. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[ https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14021: --- Status: Patch Available (was: In Progress) > When converting to CNF, fail if the expression exceeds a threshold > -- > > Key: HIVE-14021 > URL: https://issues.apache.org/jira/browse/HIVE-14021 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14021.1.patch, HIVE-14021.2.patch, HIVE-14021.patch > > > When converting to conjunctive normal form (CNF), fail if the expression > exceeds a threshold. CNF can explode exponentially in the size of the input > expression, but rarely does so in practice. Add a maxNodeCount parameter to > RexUtil.toCnf and throw or return null if it is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344422#comment-15344422
]
Aihua Xu commented on HIVE-14037:
-
+ [~Ferd] for further discussion. You worked on HIVE-7553 to add such feature.
I'm investigating the issue. Seems like we should add the jars under
hive.reloadable.aux.jars.path to tmpjars of JobConf when submitting the job to
hadoop same as hive.aux.jars.path. Otherwise, the jars in HS2 node won't be
distributed to Mapper/Reducer nodes.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13982:
---
Attachment: HIVE-13982.6.patch
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13982) Extensions to RS dedup: execute with different column order and sorting direction if possible
[
https://issues.apache.org/jira/browse/HIVE-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13982 started by Jesus Camacho Rodriguez.
--
> Extensions to RS dedup: execute with different column order and sorting
> direction if possible
> -
>
> Key: HIVE-13982
> URL: https://issues.apache.org/jira/browse/HIVE-13982
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13982.2.patch, HIVE-13982.3.patch,
> HIVE-13982.4.patch, HIVE-13982.5.patch, HIVE-13982.6.patch, HIVE-13982.patch
>
>
> Pointed out by [~gopalv].
> RS dedup should kick in for these cases, avoiding an additional shuffle stage.
> {code}
> select state, city, sum(sales) from table
> group by state, city
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state, city
> limit 10;
> {code}
> {code}
> select state, city, sum(sales) from table
> group by city, state
> order by state desc, city
> limit 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[ https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14021 started by Jesus Camacho Rodriguez. -- > When converting to CNF, fail if the expression exceeds a threshold > -- > > Key: HIVE-14021 > URL: https://issues.apache.org/jira/browse/HIVE-14021 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14021.1.patch, HIVE-14021.2.patch, HIVE-14021.patch > > > When converting to conjunctive normal form (CNF), fail if the expression > exceeds a threshold. CNF can explode exponentially in the size of the input > expression, but rarely does so in practice. Add a maxNodeCount parameter to > RexUtil.toCnf and throw or return null if it is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[ https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14021: --- Attachment: HIVE-14021.2.patch > When converting to CNF, fail if the expression exceeds a threshold > -- > > Key: HIVE-14021 > URL: https://issues.apache.org/jira/browse/HIVE-14021 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14021.1.patch, HIVE-14021.2.patch, HIVE-14021.patch > > > When converting to conjunctive normal form (CNF), fail if the expression > exceeds a threshold. CNF can explode exponentially in the size of the input > expression, but rarely does so in practice. Add a maxNodeCount parameter to > RexUtil.toCnf and throw or return null if it is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[ https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344432#comment-15344432 ] Ashutosh Chauhan commented on HIVE-14021: - +1 pending tests. > When converting to CNF, fail if the expression exceeds a threshold > -- > > Key: HIVE-14021 > URL: https://issues.apache.org/jira/browse/HIVE-14021 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14021.1.patch, HIVE-14021.2.patch, HIVE-14021.patch > > > When converting to conjunctive normal form (CNF), fail if the expression > exceeds a threshold. CNF can explode exponentially in the size of the input > expression, but rarely does so in practice. Add a maxNodeCount parameter to > RexUtil.toCnf and throw or return null if it is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14075) BeeLine.java.orig was accidentally committed during HIVE-14001 patch
[ https://issues.apache.org/jira/browse/HIVE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14075: --- Description: NO_PRECOMMIT_TEST We should remove this file. was:We should remove this file. > BeeLine.java.orig was accidentally committed during HIVE-14001 patch > > > Key: HIVE-14075 > URL: https://issues.apache.org/jira/browse/HIVE-14075 > Project: Hive > Issue Type: Bug >Reporter: Sergio Peña >Assignee: Sergio Peña > > NO_PRECOMMIT_TEST > We should remove this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344791#comment-15344791
]
Hive QA commented on HIVE-13872:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812331/HIVE-13872.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10255 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/221/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/221/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812331 - PreCommit-HIVE-MASTER-Build
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.02.patch,
> HIVE-13872.03.patch, HIVE-13872.04.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14075) BeeLine.java.orig was accidentally committed during HIVE-14001 patch
[ https://issues.apache.org/jira/browse/HIVE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14075: --- Attachment: HIVE-14075.1.patch > BeeLine.java.orig was accidentally committed during HIVE-14001 patch > > > Key: HIVE-14075 > URL: https://issues.apache.org/jira/browse/HIVE-14075 > Project: Hive > Issue Type: Bug >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14075.1.patch > > > NO_PRECOMMIT_TEST > We should remove this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14075) BeeLine.java.orig was accidentally committed during HIVE-14001 patch
[ https://issues.apache.org/jira/browse/HIVE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña resolved HIVE-14075. Resolution: Fixed Fix Version/s: 2.2.0 > BeeLine.java.orig was accidentally committed during HIVE-14001 patch > > > Key: HIVE-14075 > URL: https://issues.apache.org/jira/browse/HIVE-14075 > Project: Hive > Issue Type: Bug >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.2.0 > > Attachments: HIVE-14075.1.patch > > > NO_PRECOMMIT_TEST > We should remove this file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14063) beeline to auto connect to the HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344884#comment-15344884 ] Vihang Karajgaonkar commented on HIVE-14063: [~szehon] Can you please take a look and comment what you think about this? Thanks > beeline to auto connect to the HiveServer2 > -- > > Key: HIVE-14063 > URL: https://issues.apache.org/jira/browse/HIVE-14063 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > Currently one has to give an jdbc:hive2 url in order for Beeline to connect a > hiveserver2 instance. It would be great if Beeline can get the info somehow > (from a properties file at a well-known location?) and connect automatically > if user doesn't specify such a url. If the properties file is not present, > then beeline would expect user to provide the url and credentials using > !connect or ./beeline -u .. commands > While Beeline is flexible (being a mere JDBC client), most environments would > have just a single HS2. Having users to manually connect into this via either > "beeline ~/.propsfile" or -u or !connect statements is lowering the > experience part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Open (was: Patch Available)
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Attachment: HIVE-14070.02.patch
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-14070:
---
Status: Patch Available (was: Open)
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[ https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-14074: Attachment: HIVE-14074.01.patch > RELOAD FUNCTION should update dropped functions > --- > > Key: HIVE-14074 > URL: https://issues.apache.org/jira/browse/HIVE-14074 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Fix For: 2.2.0 > > Attachments: HIVE-14074.01.patch > > > Due to HIVE-2573, functions are stored in a per-session registry and only > loaded in from the metastore when hs2 or hive cli is started. Running RELOAD > FUNCTION in the current session is a way to force a reload of the functions, > so that changes that occurred in other running sessions will be reflected in > the current session, without having to restart the current session. However, > while functions that are created in other sessions will now appear in the > current session, functions that have been dropped are not removed from the > current session's registry. It seems inconsistent that created functions are > updated while dropped functions are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343898#comment-15343898 ] Ferdinand Xu commented on HIVE-12077: - Thanks [~chinnalalam] for the contribution. LGTM +1 pending to the test > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Attachments: HIVE-12077.1.patch, HIVE-12077.2.patch, > HIVE-12077.3.patch, HIVE-12077.4.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7044) ORC Vector: column of empty strings is read back as null
[ https://issues.apache.org/jira/browse/HIVE-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344029#comment-15344029 ] Amruth S commented on HIVE-7044: Is this issue getting contained in the newer versions of hive? I am facing this in 1.2.1 > ORC Vector: column of empty strings is read back as null > > > Key: HIVE-7044 > URL: https://issues.apache.org/jira/browse/HIVE-7044 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 0.13.0 >Reporter: Dain Sundstrom >Assignee: Jitendra Nath Pandey >Priority: Blocker > Labels: orcfile, vector > Attachments: TestOrcEmptyString.java > > > If I write a column of empty string values, the vectorized read code returns > a vector of nulls, but the non-vectorized code returns the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14021) When converting to CNF, fail if the expression exceeds a threshold
[
https://issues.apache.org/jira/browse/HIVE-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344015#comment-15344015
]
Hive QA commented on HIVE-14021:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812308/HIVE-14021.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10253 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query85
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query89
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query91
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/217/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/217/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812308 - PreCommit-HIVE-MASTER-Build
> When converting to CNF, fail if the expression exceeds a threshold
> --
>
> Key: HIVE-14021
> URL: https://issues.apache.org/jira/browse/HIVE-14021
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-14021.1.patch, HIVE-14021.patch
>
>
> When converting to conjunctive normal form (CNF), fail if the expression
> exceeds a threshold. CNF can explode exponentially in the size of the input
> expression, but rarely does so in practice. Add a maxNodeCount parameter to
> RexUtil.toCnf and throw or return null if it is exceeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14071) HIVE-14014 breaks non-file outputs
[
https://issues.apache.org/jira/browse/HIVE-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343863#comment-15343863
]
Hive QA commented on HIVE-14071:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812304/HIVE-14071.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10247 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-leftsemijoin_mr.q-bucket5.q-and-1-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/216/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/216/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812304 - PreCommit-HIVE-MASTER-Build
> HIVE-14014 breaks non-file outputs
> --
>
> Key: HIVE-14071
> URL: https://issues.apache.org/jira/browse/HIVE-14071
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14071.patch, HIVE-14071.patch
>
>
> Cannot avoid creating outputs when outputs are e.g. streaming
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344586#comment-15344586
]
Aihua Xu commented on HIVE-14037:
-
Thanks. hive.aux.jars.path uses tmpjars to pass the local jar info and then
hadoop will copy those jars to hdfs and distribute them to other nodes. Seems
like we should do the same thing. I'm investigating that and will give you
updates.
Just to confirm with you if you were aware of the issue.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9385) Sum a Double using an ORC table
[
https://issues.apache.org/jira/browse/HIVE-9385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335822#comment-15335822
]
Amruth S edited comment on HIVE-9385 at 6/22/16 11:29 AM:
--
We are using hive 1.2.1 and facing the same issue.
Is this bug occuring in hive 2.0 also?
was (Author: amrk7):
We are using hive 1.2.1 and facing the same issue. I feel this is related to
vectorisation. If I disable vectorization, I get the exact sum on a double
column.
Is this bug occuring in hive 2.0 also?
> Sum a Double using an ORC table
> ---
>
> Key: HIVE-9385
> URL: https://issues.apache.org/jira/browse/HIVE-9385
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
> Environment: HDP 2.x, Hive
>Reporter: Nick Martin
>Priority: Minor
>
> I’m storing a sales amount column as a double in an ORC table and when I do:
> {code:sql}
> select sum(x) from sometable
> {code}
> I get a value like {{4.79165141174808E9}}
> A visual inspection of the column values reveals no glaring anomalies…all
> looks pretty normal.
> If I do the same thing in a textfile table I get a perfectly fine aggregation
> of the double field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14045) (Vectorization) Add missing case for BINARY in VectorizationContext.getNormalizedName method
[
https://issues.apache.org/jira/browse/HIVE-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344159#comment-15344159
]
Hive QA commented on HIVE-14045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812320/HIVE-14045.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10251 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/218/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/218/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-218/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812320 - PreCommit-HIVE-MASTER-Build
> (Vectorization) Add missing case for BINARY in
> VectorizationContext.getNormalizedName method
>
>
> Key: HIVE-14045
> URL: https://issues.apache.org/jira/browse/HIVE-14045
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
> Fix For: 2.2.0
>
> Attachments: HIVE-14045.01.patch, HIVE-14045.02.patch,
> HIVE-14045.03.patch
>
>
> Missing case for BINARY data type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14028) stats is not updated
[
https://issues.apache.org/jira/browse/HIVE-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344321#comment-15344321
]
Hive QA commented on HIVE-14028:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812328/HIVE-14028.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10252 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbasestats
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/219/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/219/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812328 - PreCommit-HIVE-MASTER-Build
> stats is not updated
>
>
> Key: HIVE-14028
> URL: https://issues.apache.org/jira/browse/HIVE-14028
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14028.01.patch
>
>
> {code}
> DROP TABLE users;
> CREATE TABLE users(key string, state string, country string, country_id int)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> INSERT OVERWRITE TABLE users SELECT 'user1', 'IA', 'USA', 0 FROM src;
> desc formatted users;
> {code}
> the result is
> {code}
> A masked pattern was here
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles0
> numRows 0
> rawDataSize 0
> storage_handler
> org.apache.hadoop.hive.hbase.HBaseStorageHandler
> totalSize 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14045) (Vectorization) Add missing case for BINARY in VectorizationContext.getNormalizedName method
[ https://issues.apache.org/jira/browse/HIVE-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14045: Resolution: Fixed Status: Resolved (was: Patch Available) > (Vectorization) Add missing case for BINARY in > VectorizationContext.getNormalizedName method > > > Key: HIVE-14045 > URL: https://issues.apache.org/jira/browse/HIVE-14045 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline > Fix For: 2.2.0 > > Attachments: HIVE-14045.01.patch, HIVE-14045.02.patch, > HIVE-14045.03.patch > > > Missing case for BINARY data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14045) (Vectorization) Add missing case for BINARY in VectorizationContext.getNormalizedName method
[ https://issues.apache.org/jira/browse/HIVE-14045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344330#comment-15344330 ] Matt McCline commented on HIVE-14045: - Committed to master. > (Vectorization) Add missing case for BINARY in > VectorizationContext.getNormalizedName method > > > Key: HIVE-14045 > URL: https://issues.apache.org/jira/browse/HIVE-14045 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline > Fix For: 2.2.0 > > Attachments: HIVE-14045.01.patch, HIVE-14045.02.patch, > HIVE-14045.03.patch > > > Missing case for BINARY data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14057) Add an option in llapstatus to generate output to a file
[ https://issues.apache.org/jira/browse/HIVE-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344919#comment-15344919 ] Siddharth Seth commented on HIVE-14057: --- Test failures are unrelated. Committing. Thanks for the review [~sershe] > Add an option in llapstatus to generate output to a file > > > Key: HIVE-14057 > URL: https://issues.apache.org/jira/browse/HIVE-14057 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14057.01.patch, HIVE-14057.02.patch, > HIVE-14057.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: (was: proposal.pdf) > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: proposal.pdf > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13380) Decimal should have lower precedence than double in type hierachy
[ https://issues.apache.org/jira/browse/HIVE-13380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344968#comment-15344968 ] Sergey Shelukhin commented on HIVE-13380: - What is the alternative course of action that you propose? This patch breaks ANSI compat on at least some queries and causes incorrect results, including for e.g. TPCH which is an industry benchmark. I think we should revert it, then after adding some tests and making sure they pass, commit it back. > Decimal should have lower precedence than double in type hierachy > - > > Key: HIVE-13380 > URL: https://issues.apache.org/jira/browse/HIVE-13380 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-13380.2.patch, HIVE-13380.4.patch, > HIVE-13380.5.patch, HIVE-13380.patch, decimal_filter.q > > > Currently its other way round. Also, decimal should be lower than float. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344982#comment-15344982
]
Sergey Shelukhin commented on HIVE-14070:
-
Does TIME_TO_SUBMIT need to be removed? Otherwise lgtm. cc [~prasanth_j]
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14057) Add an option in llapstatus to generate output to a file
[ https://issues.apache.org/jira/browse/HIVE-14057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14057: -- Resolution: Fixed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) > Add an option in llapstatus to generate output to a file > > > Key: HIVE-14057 > URL: https://issues.apache.org/jira/browse/HIVE-14057 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.1 > > Attachments: HIVE-14057.01.patch, HIVE-14057.02.patch, > HIVE-14057.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14076) "Vectorizaton is not supported for datatype:VOID" error while inserting data into specific columns
[
https://issues.apache.org/jira/browse/HIVE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14076:
---
Status: Patch Available (was: In Progress)
> "Vectorizaton is not supported for datatype:VOID" error while inserting data
> into specific columns
> --
>
> Key: HIVE-14076
> URL: https://issues.apache.org/jira/browse/HIVE-14076
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jagruti Varia
>Assignee: Jesus Camacho Rodriguez
>
> Insert into specific columns fails due to following error:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1465261180142_0160_1_01,
> diagnostics=[Task failed, taskId=task_1465261180142_0160_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1465261180142_0160_1_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:221)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.initializeSourceForTag(ReduceRecordProcessor.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:163)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: java.lang.RuntimeException: Vectorizaton is not supported for
> datatype:VOID
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.createColumnVector(VectorizedBatchUtil.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.createVectorizedRowBatch(VectorizedRowBatchCtx.java:194)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:177)
> ... 17 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1465261180142_0160_1_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Updated] (HIVE-14076) "Vectorizaton is not supported for datatype:VOID" error while inserting data into specific columns
[
https://issues.apache.org/jira/browse/HIVE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14076:
---
Attachment: HIVE-14076.patch
[~ekoifman], could you review it? Thanks
> "Vectorizaton is not supported for datatype:VOID" error while inserting data
> into specific columns
> --
>
> Key: HIVE-14076
> URL: https://issues.apache.org/jira/browse/HIVE-14076
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jagruti Varia
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14076.patch
>
>
> Insert into specific columns fails due to following error:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1465261180142_0160_1_01,
> diagnostics=[Task failed, taskId=task_1465261180142_0160_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1465261180142_0160_1_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:221)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.initializeSourceForTag(ReduceRecordProcessor.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:163)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: java.lang.RuntimeException: Vectorizaton is not supported for
> datatype:VOID
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.createColumnVector(VectorizedBatchUtil.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.createVectorizedRowBatch(VectorizedRowBatchCtx.java:194)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:177)
> ... 17 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1465261180142_0160_1_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Commented] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[ https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344975#comment-15344975 ] Sergey Shelukhin commented on HIVE-14074: - As far as I see unregisterPermanentFunctions goes to system registry, not the session one. Also, the update may fail, resulting in all functions being removed. I think the removal should be integrated into the update, by e.g. taking the existing set, then removing stuff from it as we process the new list; whatever in the set can then be removed. > RELOAD FUNCTION should update dropped functions > --- > > Key: HIVE-14074 > URL: https://issues.apache.org/jira/browse/HIVE-14074 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Fix For: 2.2.0 > > Attachments: HIVE-14074.01.patch > > > Due to HIVE-2573, functions are stored in a per-session registry and only > loaded in from the metastore when hs2 or hive cli is started. Running RELOAD > FUNCTION in the current session is a way to force a reload of the functions, > so that changes that occurred in other running sessions will be reflected in > the current session, without having to restart the current session. However, > while functions that are created in other sessions will now appear in the > current session, functions that have been dropped are not removed from the > current session's registry. It seems inconsistent that created functions are > updated while dropped functions are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[ https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344987#comment-15344987 ] Abdullah Yousufi commented on HIVE-14074: - Ah that's a good point. I had considered the set removal method, but was curious if this one-liner would work. I'll implement it that way instead. > RELOAD FUNCTION should update dropped functions > --- > > Key: HIVE-14074 > URL: https://issues.apache.org/jira/browse/HIVE-14074 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Fix For: 2.2.0 > > Attachments: HIVE-14074.01.patch > > > Due to HIVE-2573, functions are stored in a per-session registry and only > loaded in from the metastore when hs2 or hive cli is started. Running RELOAD > FUNCTION in the current session is a way to force a reload of the functions, > so that changes that occurred in other running sessions will be reflected in > the current session, without having to restart the current session. However, > while functions that are created in other sessions will now appear in the > current session, functions that have been dropped are not removed from the > current session's registry. It seems inconsistent that created functions are > updated while dropped functions are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344992#comment-15344992
]
Sergey Shelukhin commented on HIVE-14070:
-
Does TIME_TO_SUBMIT need to be removed? Otherwise lgtm. cc [~prasanth_j]
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13744) LLAP IO - add complex types support
[ https://issues.apache.org/jira/browse/HIVE-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13744: - Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Test failures are unrelated and are happening for other patches as well. Committed patch to branch-2.1 and master. Thanks [~sershe] for the review! > LLAP IO - add complex types support > --- > > Key: HIVE-13744 > URL: https://issues.apache.org/jira/browse/HIVE-13744 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Labels: llap, orc > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-13744.1.patch, HIVE-13744.2.patch > > > Recently, complex type column vectors were added to Hive. We should use them > in IO elevator. > Vectorization itself doesn't support complex types (yet), but this would be > useful when it does, also it will enable LLAP IO elevator to be used in > non-vectorized context with complex types after HIVE-13617 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13159: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Patch 3 committed. I also remove the TxnHandler.java.orig file which should not have been there. > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Fix For: 2.2.0 > > Attachments: HIVE-13159.2.patch, HIVE-13159.3.patch, HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14076) "Vectorizaton is not supported for datatype:VOID" error while inserting data into specific columns
[
https://issues.apache.org/jira/browse/HIVE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14076 started by Jesus Camacho Rodriguez.
--
> "Vectorizaton is not supported for datatype:VOID" error while inserting data
> into specific columns
> --
>
> Key: HIVE-14076
> URL: https://issues.apache.org/jira/browse/HIVE-14076
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jagruti Varia
>Assignee: Jesus Camacho Rodriguez
>
> Insert into specific columns fails due to following error:
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1465261180142_0160_1_01,
> diagnostics=[Task failed, taskId=task_1465261180142_0160_1_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1465261180142_0160_1_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:221)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.initializeSourceForTag(ReduceRecordProcessor.java:245)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:163)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> ... 14 more
> Caused by: java.lang.RuntimeException: Vectorizaton is not supported for
> datatype:VOID
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.createColumnVector(VectorizedBatchUtil.java:172)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.createVectorizedRowBatch(VectorizedRowBatchCtx.java:194)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.init(ReduceRecordSource.java:177)
> ... 17 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1465261180142_0160_1_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: Reduce operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344663#comment-15344663
]
Pengcheng Xiong commented on HIVE-14070:
[~ashutoshc] and [~sershe]. I also addressed some other problems which are
related with the "hive.tez.exec.print.summary=true". (1) original code L340 in
Driver.java, "PerfLogger perfLogger = SessionState.getPerfLogger();" this
should be reset otherwise the same PerfLogger will be reused across different
queries from the same session in CLI. (2) I removed "PerfLogger.TIME_TO_SUBMIT"
as it is never used. (3) I'm thinking about removing PerfLogger.DRIVER_RUN as
well. Right now it is used to denote the query start time. I would suggest use
the start time of PerfLogger.COMPILE instead.
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both
[ https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14001: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~pvary]. I committed this to master. The test is not related to this patch. I run it again without the patch and it failed as well. Seems it is an issue on my environment. > beeline doesn't give out an error when takes either "-e" or "-f" in command > instead of both > --- > > Key: HIVE-14001 > URL: https://issues.apache.org/jira/browse/HIVE-14001 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 0.10.0, 2.0.1 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Trivial > Fix For: 2.2.0 > > Attachments: HIVE-14001.2.patch, HIVE-14001.patch > > > When providing both arguments there should be an error message -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[
https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344618#comment-15344618
]
Sergio Peña commented on HIVE-14074:
[~ayousufi] is this thread-safe? I see that
{{FunctionRegistry.unregisterPermanentFunctions(}} has a lock/unlock block
while the functions are deleted, but what would happen while functions are
added back? During the time where functions are added, then we do not have any
function on the DB, will other users have issues with queries that use
functions that were deleted for a short period of time?
> RELOAD FUNCTION should update dropped functions
> ---
>
> Key: HIVE-14074
> URL: https://issues.apache.org/jira/browse/HIVE-14074
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
> Fix For: 2.2.0
>
> Attachments: HIVE-14074.01.patch
>
>
> Due to HIVE-2573, functions are stored in a per-session registry and only
> loaded in from the metastore when hs2 or hive cli is started. Running RELOAD
> FUNCTION in the current session is a way to force a reload of the functions,
> so that changes that occurred in other running sessions will be reflected in
> the current session, without having to restart the current session. However,
> while functions that are created in other sessions will now appear in the
> current session, functions that have been dropped are not removed from the
> current session's registry. It seems inconsistent that created functions are
> updated while dropped functions are not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong results on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345197#comment-15345197
]
Thejas M Nair commented on HIVE-14070:
--
Please remove the unnecessary call to 'SessionState.getPerfLogger();' I
mentioned earlier as well. (or clarify why it would be needed).
> hive.tez.exec.print.summary=true returns wrong results on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14028) stats is not updated
[
https://issues.apache.org/jira/browse/HIVE-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345222#comment-15345222
]
Ashutosh Chauhan commented on HIVE-14028:
-
+1
> stats is not updated
>
>
> Key: HIVE-14028
> URL: https://issues.apache.org/jira/browse/HIVE-14028
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14028.01.patch, HIVE-14028.02.patch
>
>
> {code}
> DROP TABLE users;
> CREATE TABLE users(key string, state string, country string, country_id int)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> INSERT OVERWRITE TABLE users SELECT 'user1', 'IA', 'USA', 0 FROM src;
> desc formatted users;
> {code}
> the result is
> {code}
> A masked pattern was here
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles0
> numRows 0
> rawDataSize 0
> storage_handler
> org.apache.hadoop.hive.hbase.HBaseStorageHandler
> totalSize 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row
[
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HIVE-13946.
-
Resolution: Duplicate
The same fix covers this
> Decimal value need to be single-quoted when selecting where clause with that
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values
> (4327269606205.029297);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464727816747_0762)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=1, totalSize=21,
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> 4327269606205.029297;
> +--+--+
> | test.dc |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test
> where dc = 4327269606205.029297;
> +--+--+
> | Explain|
> +--+--+
> | STAGE DEPENDENCIES: |
> | Stage-0 is a root stage|
> | |
> | STAGE PLANS: |
> | Stage: Stage-0 |
> | Fetch Operator |
> | limit: -1 |
> | Processor Tree:|
> | TableScan|
> | alias: test|
> | filterExpr: (dc = 4.3272696062050293E12) (type: boolean) |
> | Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean) |
> | Select Operator |
> | expressions: dc (type: decimal(38,18)) |
> | outputColumnNames: _col0 |
> | ListSink |
> | |
> +--+--+
> 18 rows selected (0.512 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row
[
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13946:
Fix Version/s: (was: 1.2.1)
> Decimal value need to be single-quoted when selecting where clause with that
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values
> (4327269606205.029297);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464727816747_0762)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=1, totalSize=21,
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> 4327269606205.029297;
> +--+--+
> | test.dc |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test
> where dc = 4327269606205.029297;
> +--+--+
> | Explain|
> +--+--+
> | STAGE DEPENDENCIES: |
> | Stage-0 is a root stage|
> | |
> | STAGE PLANS: |
> | Stage: Stage-0 |
> | Fetch Operator |
> | limit: -1 |
> | Processor Tree:|
> | TableScan|
> | alias: test|
> | filterExpr: (dc = 4.3272696062050293E12) (type: boolean) |
> | Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean) |
> | Select Operator |
> | expressions: dc (type: decimal(38,18)) |
> | outputColumnNames: _col0 |
> | ListSink |
> | |
> +--+--+
> 18 rows selected (0.512 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13723:
---
Status: Open (was: Patch Available)
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13723:
---
Attachment: (was: HIVE-13723.2.patch)
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13723:
---
Attachment: (was: HIVE-13723.1.patch)
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
[jira] [Updated] (HIVE-11832) HIVE-11802 breaks compilation in JDK 8
[ https://issues.apache.org/jira/browse/HIVE-11832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11832: Fix Version/s: 1.2.2 > HIVE-11802 breaks compilation in JDK 8 > -- > > Key: HIVE-11832 > URL: https://issues.apache.org/jira/browse/HIVE-11832 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Sergio Peña > Fix For: 1.3.0, 2.0.0, 1.2.2 > > Attachments: HIVE-11832.1.patch > > > HIVE-11802 changes breaks JDK 8 compilation. FloatingDecimal constructor > accepting float is removed in JDK 8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11802) Float-point numbers are displayed with different precision in Beeline/JDBC
[
https://issues.apache.org/jira/browse/HIVE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345410#comment-15345410
]
Sergey Shelukhin commented on HIVE-11802:
-
Backported both commits to 1.3 and 1.2.2
> Float-point numbers are displayed with different precision in Beeline/JDBC
> --
>
> Key: HIVE-11802
> URL: https://issues.apache.org/jira/browse/HIVE-11802
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Sergio Peña
>Assignee: lijp
> Fix For: 1.3.0, 2.0.0, 1.2.2
>
> Attachments: HIVE-11802.3.patch
>
>
> When inserting float-point numbers to a table, the values displayed on
> beeline or jdbc are with different precision.
> How to reproduce:
> {noformat}
> 0: jdbc:hive2://localhost:1> create table decimals (f float, af
> array, d double, ad array) stored as parquet;
> No rows affected (0.294 seconds)
> 0: jdbc:hive2://localhost:1> insert into table decimals select 1.10058,
> array(cast(1.10058 as float)), 2.0133, array(2.0133) from dummy limit 1;
> ...
> No rows affected (20.089 seconds)
> 0: jdbc:hive2://localhost:1> select f, af, af[0], d, ad[0] from decimals;
> +-++-+-+-+--+
> | f | af | _c2 |d| _c4 |
> +-++-+-+-+--+
> | 1.1005799770355225 | [1.10058] | 1.1005799770355225 | 2.0133 | 2.0133 |
> +-++-+-+-+--+
> {noformat}
> When displaying arrays, the values are displayed correctly, but if I print a
> specific element, it is then displayed with more decimal positions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11802) Float-point numbers are displayed with different precision in Beeline/JDBC
[
https://issues.apache.org/jira/browse/HIVE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-11802:
Fix Version/s: 1.2.2
1.3.0
> Float-point numbers are displayed with different precision in Beeline/JDBC
> --
>
> Key: HIVE-11802
> URL: https://issues.apache.org/jira/browse/HIVE-11802
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Sergio Peña
>Assignee: lijp
> Fix For: 1.3.0, 2.0.0, 1.2.2
>
> Attachments: HIVE-11802.3.patch
>
>
> When inserting float-point numbers to a table, the values displayed on
> beeline or jdbc are with different precision.
> How to reproduce:
> {noformat}
> 0: jdbc:hive2://localhost:1> create table decimals (f float, af
> array, d double, ad array) stored as parquet;
> No rows affected (0.294 seconds)
> 0: jdbc:hive2://localhost:1> insert into table decimals select 1.10058,
> array(cast(1.10058 as float)), 2.0133, array(2.0133) from dummy limit 1;
> ...
> No rows affected (20.089 seconds)
> 0: jdbc:hive2://localhost:1> select f, af, af[0], d, ad[0] from decimals;
> +-++-+-+-+--+
> | f | af | _c2 |d| _c4 |
> +-++-+-+-+--+
> | 1.1005799770355225 | [1.10058] | 1.1005799770355225 | 2.0133 | 2.0133 |
> +-++-+-+-+--+
> {noformat}
> When displaying arrays, the values are displayed correctly, but if I print a
> specific element, it is then displayed with more decimal positions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: (was: proposal.pdf) > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[
https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345781#comment-15345781
]
Colin Patrick McCabe edited comment on HIVE-13966 at 6/23/16 5:30 AM:
--
I definitely agree that a synchronous listener is what we need.
It seems like the listener type itself should know whether it needs to be
synchronous or not, right? We can have a method somewhere in
{{MetaStoreEventListener}}, which defaults to false, which specifies whether a
listener is synchronous or not. There shouldn't be any need to configure this
since we should already know for each listener whether it needs to be
synchronous or not based on its type.
was (Author: cmccabe):
I definitely agree that a synchronous listener is what we need.
It seems like the listener type itself should know whether it needs to be
synchronous or not, right? We can have a method somewhere in base class or
interface that specifies whether a listener is synchronous or not. There
shouldn't be any need to configure this since we should already know for each
listener whether it needs to be synchronous or not based on its type.
> DbNotificationListener: can loose DDL operation notifications
> -
>
> Key: HIVE-13966
> URL: https://issues.apache.org/jira/browse/HIVE-13966
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Nachiket Vaidya
>Assignee: Rahul Sharma
>Priority: Critical
>
> The code for each API in HiveMetaStore.java is like this:
> 1. openTransaction()
> 2. -- operation--
> 3. commit() or rollback() based on result of the operation.
> 4. add entry to notification log (unconditionally)
> If the operation is failed (in step 2), we still add entry to notification
> log. Found this issue in testing.
> It is still ok as this is the case of false positive.
> If the operation is successful and adding to notification log failed, the
> user will get an MetaException. It will not rollback the operation, as it is
> already committed. We need to handle this case so that we will not have false
> negatives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: proposal.pdf > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: proposal.pdf > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7443) Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs
[
https://issues.apache.org/jira/browse/HIVE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345788#comment-15345788
]
Hive QA commented on HIVE-7443:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812483/HIVE-7443.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10254 tests
executed
*Failed tests:*
{noformat}
TestSchedulerQueue - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-227/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812483 - PreCommit-HIVE-MASTER-Build
> Fix HiveConnection to communicate with Kerberized Hive JDBC server and
> alternative JDKs
> ---
>
> Key: HIVE-7443
> URL: https://issues.apache.org/jira/browse/HIVE-7443
> Project: Hive
> Issue Type: Bug
> Components: JDBC, Security
>Affects Versions: 0.12.0, 0.13.1
> Environment: Kerberos
> Run Hive server2 and client with IBM JDK7.1
>Reporter: Yu Gao
>Assignee: Aihua Xu
> Attachments: HIVE-7443.2.patch, HIVE-7443.3.patch, HIVE-7443.patch
>
>
> Hive Kerberos authentication has been enabled in my cluster. I ran kinit to
> initialize the current login user's ticket cache successfully, and then tried
> to use beeline to connect to Hive Server2, but failed. After I manually added
> some logging to catch the failure exception, this is what I got that caused
> the failure:
> beeline> !connect
> jdbc:hive2://:1/default;principal=hive/@REALM.COM
> org.apache.hive.jdbc.HiveDriver
> scan complete in 2ms
> Connecting to
> jdbc:hive2://:1/default;principal=hive/@REALM.COM
> Enter password for
> jdbc:hive2://:1/default;principal=hive/@REALM.COM:
> 14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport
> javax.security.sasl.SaslException: Failed to open client transport [Caused by
> java.io.IOException: Could not instantiate SASL transport]
> at
> org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
> at java.sql.DriverManager.getConnection(DriverManager.java:582)
> at java.sql.DriverManager.getConnection(DriverManager.java:198)
> at
> org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
> at
> org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186)
> at org.apache.hive.beeline.Commands.connect(Commands.java:959)
> at org.apache.hive.beeline.Commands.connect(Commands.java:880)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:619)
> at
> org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345781#comment-15345781 ] Colin Patrick McCabe commented on HIVE-13966: - I definitely agree that a synchronous listener is what we need. It seems like the listener type itself should know whether it needs to be synchronous or not, right? We can have a method somewhere in base class or interface that specifies whether a listener is synchronous or not. There shouldn't be any need to configure this since we should already know for each listener whether it needs to be synchronous or not based on its type. > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14077) revert or fix HIVE-13380
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345221#comment-15345221 ] Sergey Shelukhin commented on HIVE-14077: - Heh, I am about to do that as part of some other bugfix > revert or fix HIVE-13380 > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Blocker > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: HIVE-13945.patch
This changes Hive to prefer decimal over float for literals. [~ashutoshc] can
you take a look? This might cause some q file changes, so it will probably
require an update
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: HIVE-13945.patch
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14063) beeline to auto connect to the HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345408#comment-15345408 ] Szehon Ho commented on HIVE-14063: -- And one question to think about is , if you override 'url' from the command line, will beeline prompt you for new username, password, as its a new HS2 HA group/instance. > beeline to auto connect to the HiveServer2 > -- > > Key: HIVE-14063 > URL: https://issues.apache.org/jira/browse/HIVE-14063 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > Currently one has to give an jdbc:hive2 url in order for Beeline to connect a > hiveserver2 instance. It would be great if Beeline can get the info somehow > (from a properties file at a well-known location?) and connect automatically > if user doesn't specify such a url. If the properties file is not present, > then beeline would expect user to provide the url and credentials using > !connect or ./beeline -u .. commands > While Beeline is flexible (being a mere JDBC client), most environments would > have just a single HS2. Having users to manually connect into this via either > "beeline ~/.propsfile" or -u or !connect statements is lowering the > experience part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14079) Remove file, method and line number from pattern layout
[
https://issues.apache.org/jira/browse/HIVE-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14079:
-
Attachment: HIVE-14079.1.patch
[~sershe] Can you plz review this small patch?
> Remove file, method and line number from pattern layout
> ---
>
> Key: HIVE-14079
> URL: https://issues.apache.org/jira/browse/HIVE-14079
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14079.1.patch
>
>
> Using %F%M and %L in pattern layouts need location information which is
> expensive to get and is disabled by default. We should remove them from the
> default layouts. This will avoid creating empty brackets like below
> {code}
> lockmgr.DbTxnManager (:())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Attachment: (was: HIVE-13988.02.patch)
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14063) beeline to auto connect to the HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345404#comment-15345404 ] Szehon Ho commented on HIVE-14063: -- I think it will be very useful, to have a properties file to allow for auto specification of properties like url, name, password. I assume that's the properties that are proposed, or is there more? Another idea is to detect a local HS2 and connect to that, but it would be orthogonal. Some thoughts on this proposal: I'm not sure about the need for a --property-file if user can override some properties via the command line. A lot of times there is more than one HS2 in a cluster, like HA Proxy or ZK quorum load balancing. I assume the proposal also allows for connecting to those URL's as well, to get a random HS2? And also for kerberos as well? One usability issue is that those beeline properties are defined by just a letter like 'u', 'n', 'p'. Hence it would be confusing in the properties file if the key is different, maybe its better if beeline itself took more descriptive names in addition to letters to make it consistent. > beeline to auto connect to the HiveServer2 > -- > > Key: HIVE-14063 > URL: https://issues.apache.org/jira/browse/HIVE-14063 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > Currently one has to give an jdbc:hive2 url in order for Beeline to connect a > hiveserver2 instance. It would be great if Beeline can get the info somehow > (from a properties file at a well-known location?) and connect automatically > if user doesn't specify such a url. If the properties file is not present, > then beeline would expect user to provide the url and credentials using > !connect or ./beeline -u .. commands > While Beeline is flexible (being a mere JDBC client), most environments would > have just a single HS2. Having users to manually connect into this via either > "beeline ~/.propsfile" or -u or !connect statements is lowering the > experience part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14079) Remove file, method and line number from pattern layout
[
https://issues.apache.org/jira/browse/HIVE-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14079:
-
Status: Patch Available (was: Open)
> Remove file, method and line number from pattern layout
> ---
>
> Key: HIVE-14079
> URL: https://issues.apache.org/jira/browse/HIVE-14079
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14079.1.patch
>
>
> Using %F%M and %L in pattern layouts need location information which is
> expensive to get and is disabled by default. We should remove them from the
> default layouts. This will avoid creating empty brackets like below
> {code}
> lockmgr.DbTxnManager (:())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: (was: HIVE-13945.patch)
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Status: Patch Available (was: Open)
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch, HIVE-13988.02.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Attachment: HIVE-13988.02.patch
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch, HIVE-13988.02.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13723:
---
Attachment: HIVE-13723.2.patch.txt
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
> Attachments: HIVE-13723.2.patch.txt
>
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>
[jira] [Updated] (HIVE-13723) Executing join query on type Float using Thrift Serde will result in Float cast to Double error
[
https://issues.apache.org/jira/browse/HIVE-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziyang Zhao updated HIVE-13723:
---
Status: Patch Available (was: Open)
> Executing join query on type Float using Thrift Serde will result in Float
> cast to Double error
> ---
>
> Key: HIVE-13723
> URL: https://issues.apache.org/jira/browse/HIVE-13723
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC, Serializers/Deserializers
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
>Priority: Critical
> Attachments: HIVE-13723.2.patch.txt
>
>
> After enable thrift Serde, execute the following queries in beeline,
> >create table test1 (a int);
> >create table test2 (b float);
> >insert into test1 values (1);
> >insert into test2 values (1);
> >select * from test1 join test2 on test1.a=test2.b;
> this will give the error:
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:168)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"b":1.0}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:159)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_95]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_95]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_95]
> at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_95]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.ClassCastException:
> java.lang.Float cannot be cast to java.lang.Double
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
>
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Status: Open (was: Patch Available)
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch, HIVE-13988.02.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Status: Patch Available (was: Open)
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[
https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345401#comment-15345401
]
Hive QA commented on HIVE-13878:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812360/HIVE-13878.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10257 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/224/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/224/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812360 - PreCommit-HIVE-MASTER-Build
> Vectorization: Column pruning for Text vectorization
>
>
> Key: HIVE-13878
> URL: https://issues.apache.org/jira/browse/HIVE-13878
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13878.04.patch, HIVE-13878.1.patch,
> HIVE-13878.2.patch, HIVE-13878.3.patch
>
>
> Column pruning in TextFile vectorization does not work with Vector SerDe
> settings due to LazySimple deser codepath issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13988:
---
Attachment: HIVE-13988.02.patch
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch, HIVE-13988.02.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14077) revert or fix HIVE-13380
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345212#comment-15345212 ] Ashutosh Chauhan commented on HIVE-14077: - I have noted this on HIVE-13380, pasting it here again on course of action I am suggesting: bq. I think correct course of action here is to treat literal constants as big decimal instead of current choice of double. That will get us consistent behavior. I checked on MySQL and its doing the same thing. > revert or fix HIVE-13380 > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Blocker > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13965) Empty resultset run into Exception when using Thrift Binary Serde
[
https://issues.apache.org/jira/browse/HIVE-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345314#comment-15345314
]
Vaibhav Gumashta commented on HIVE-13965:
-
+1 pending unit test analysis.
> Empty resultset run into Exception when using Thrift Binary Serde
> -
>
> Key: HIVE-13965
> URL: https://issues.apache.org/jira/browse/HIVE-13965
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Ziyang Zhao
>Assignee: Ziyang Zhao
> Attachments: HIVE-13965.1.patch.txt
>
>
> This error can be reproduced by enabling thrift binary serde, using beeline
> connect to hiveserver2 and executing the following commands:
> >create table test3(num1 int);
> >create table test4(num1 int);
> >insert into test3 values(1);
> >insert into test4 values(2);
> >select * from test3 join test4 on test3.num1=test4.num1;
> The result should be empty, but it gives an exception:
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1029)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:641)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:655)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:195)
> ... 8 more
> This error is caused in FileSinkOperator.java.
> If the resultset is empty, function process() will not be called, so variable
> "fpaths" will not be set. When run into CloseOp(),
> if (conf.isHiveServerQuery() && HiveConf.getBoolVar(hconf,
> HiveConf.ConfVars.HIVE_SERVER2_THRIFT_RESULTSET_SERIALIZE_IN_TASKS)
> &&
>
> serializer.getClass().getName().equalsIgnoreCase(ThriftJDBCBinarySerDe.class.getName()))
> {
> try {
>recordValue = serializer.serialize(null, inputObjInspectors[0]);
>rowOutWriters = fpaths.outWriters;
>rowOutWriters[0].write(recordValue);
> } catch (SerDeException | IOException e) {
>throw new HiveException(e);
> }
> }
> Here fpaths is null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Target Version/s: 1.3.0, 2.2.0, 2.1.1, 2.0.2 (was: 2.2.0, 2.1.1)
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345382#comment-15345382
]
Matt McCline commented on HIVE-13872:
-
Failures look unrelated.
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.02.patch,
> HIVE-13872.03.patch, HIVE-13872.04.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13872) Vectorization: Fix cross-product reduce sink serialization
[
https://issues.apache.org/jira/browse/HIVE-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345439#comment-15345439
]
Gopal V commented on HIVE-13872:
LGTM - +1.
minor nit
{code}
+ColumnVector colVector = batch.cols[i];
+if (colVector != null) {
+ batch.cols[i].reset();
+ batch.cols[i].ensureSize((int) batchSize, false);
+ fields[i].nextVector(batch.cols[i], null, batchSize);
+}
{code}
the code inside the condition is clearer if it uses the condition variable.
> Vectorization: Fix cross-product reduce sink serialization
> --
>
> Key: HIVE-13872
> URL: https://issues.apache.org/jira/browse/HIVE-13872
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Matt McCline
> Attachments: HIVE-13872.01.patch, HIVE-13872.02.patch,
> HIVE-13872.03.patch, HIVE-13872.04.patch, HIVE-13872.WIP.patch,
> customer_demographics.txt, vector_include_no_sel.q,
> vector_include_no_sel.q.out
>
>
> TPC-DS Q13 produces a cross-product without CBO simplifying the query
> {code}
> Caused by: java.lang.RuntimeException: null STRING entry: batchIndex 0
> projection column num 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.nullBytesReadError(VectorExtractRow.java:349)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRowColumn(VectorExtractRow.java:267)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.extractRow(VectorExtractRow.java:343)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:103)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> {code}
> Simplified query
> {code}
> set hive.cbo.enable=false;
> -- explain
> select count(1)
> from store_sales
> ,customer_demographics
> where (
> (
> customer_demographics.cd_demo_sk = store_sales.ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'M'
> )or
> (
>customer_demographics.cd_demo_sk = ss_cdemo_sk
> and customer_demographics.cd_marital_status = 'U'
> ))
> ;
> {code}
> {code}
> Map 3
> Map Operator Tree:
> TableScan
> alias: customer_demographics
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1920800 Data size: 717255532 Basic
> stats: COMPLETE Column stats: NONE
> value expressions: cd_demo_sk (type: int),
> cd_marital_status (type: string)
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14055) directSql - getting the number of partitions is broken
[
https://issues.apache.org/jira/browse/HIVE-14055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345207#comment-15345207
]
Hive QA commented on HIVE-14055:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812350/HIVE-14055.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10257 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/223/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/223/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812350 - PreCommit-HIVE-MASTER-Build
> directSql - getting the number of partitions is broken
> --
>
> Key: HIVE-14055
> URL: https://issues.apache.org/jira/browse/HIVE-14055
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14055.01.patch, HIVE-14055.02.patch,
> HIVE-14055.patch
>
>
> Noticed while looking at something else. If the filter cannot be pushed down
> it just returns 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14070) hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
[
https://issues.apache.org/jira/browse/HIVE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-14070:
-
Summary: hive.tez.exec.print.summary=true returns wrong performance numbers
on HS2 (was: hive.tez.exec.print.summary=true returns wrong results on HS2)
> hive.tez.exec.print.summary=true returns wrong performance numbers on HS2
> -
>
> Key: HIVE-14070
> URL: https://issues.apache.org/jira/browse/HIVE-14070
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14070.01.patch, HIVE-14070.02.patch
>
>
> On master, we have
> {code}
> Query Execution Summary
> --
> OPERATIONDURATION
> --
> Compile Query -1466208820.74s
> Prepare Plan0.00s
> Submit Plan 1466208825.50s
> Start DAG 0.26s
> Run DAG 4.39s
> --
> Task Execution Summary
> --
> VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
> OUTPUT_RECORDS
> --
> Map 11014.00 1,534 11 1,500
> 1
> Reducer 2 96.00 5410 1
> 0
> --
> {code}
> sounds like a real issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14074) RELOAD FUNCTION should update dropped functions
[ https://issues.apache.org/jira/browse/HIVE-14074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-14074: Attachment: HIVE-14074.02.patch > RELOAD FUNCTION should update dropped functions > --- > > Key: HIVE-14074 > URL: https://issues.apache.org/jira/browse/HIVE-14074 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi > Fix For: 2.2.0 > > Attachments: HIVE-14074.01.patch, HIVE-14074.02.patch > > > Due to HIVE-2573, functions are stored in a per-session registry and only > loaded in from the metastore when hs2 or hive cli is started. Running RELOAD > FUNCTION in the current session is a way to force a reload of the functions, > so that changes that occurred in other running sessions will be reflected in > the current session, without having to restart the current session. However, > while functions that are created in other sessions will now appear in the > current session, functions that have been dropped are not removed from the > current session's registry. It seems inconsistent that created functions are > updated while dropped functions are not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9756) LLAP: use log4j 2 for llap (log to separate files, etc.)
[ https://issues.apache.org/jira/browse/HIVE-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-9756: Attachment: HIVE-9756.5.patch This patch needs HIVE-13258 for setting up MDC in thread pool. > LLAP: use log4j 2 for llap (log to separate files, etc.) > > > Key: HIVE-9756 > URL: https://issues.apache.org/jira/browse/HIVE-9756 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran > Attachments: HIVE-9756.1.patch, HIVE-9756.2.patch, HIVE-9756.3.patch, > HIVE-9756.4.patch, HIVE-9756.4.patch, HIVE-9756.5.patch > > > For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get > throughput friendly logging. > http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13988) zero length file is being created for empty bucket in tez mode (I)
[
https://issues.apache.org/jira/browse/HIVE-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345425#comment-15345425
]
Pengcheng Xiong edited comment on HIVE-13988 at 6/22/16 11:45 PM:
--
[~ashutoshc], your comments are valid. Could u take another look? I tried to
only use move task but it seems more complicated than i thought. Move task is
followed by stats task and we also need to make stats work. Thus, I only make
very limited optimization, i.e., when there is only one "insert into", we skip
the task compilation. Please see attached q files for examples. Thanks.
was (Author: pxiong):
[~ashutoshc], your comments are valid. Could u take another look? I tried to
only use move task but it seems more complicated than i thought. Move task is
followed by stats task and we also need to make stats work. Thus, I only make
very limited optimization, i.e., when there is only one "insert into", we skip
the task compilation. Please see attached q files for examples.
> zero length file is being created for empty bucket in tez mode (I)
> --
>
> Key: HIVE-13988
> URL: https://issues.apache.org/jira/browse/HIVE-13988
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13988.01.patch, HIVE-13988.02.patch
>
>
> Even though bucket is empty, zero length file is being created in tez mode.
> steps to reproduce the issue:
> {noformat}
> hive> set hive.execution.engine;
> hive.execution.engine=tez
> hive> drop table if exists emptybucket_orc;
> OK
> Time taken: 5.416 seconds
> hive> create table emptybucket_orc(age int) clustered by (age) sorted by
> (age) into 99 buckets stored as orc;
> OK
> Time taken: 0.493 seconds
> hive> insert into table emptybucket_orc select distinct(age) from
> studenttab10k limit 0;
> Query ID = hrt_qa_20160523231955_8b981be7-68c4-4416-8a48-5f8c7ff551c3
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1464045121842_0002)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 2 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 3 .. llap SUCCEEDED 1 100
> 0 0
> Reducer 4 .. llap SUCCEEDED 99 9900
> 0 0
> --
> VERTICES: 04/04 [==>>] 100% ELAPSED TIME: 11.00 s
>
> --
> Loading data to table default.emptybucket_orc
> OK
> Time taken: 16.907 seconds
> hive> dfs -ls /apps/hive/warehouse/emptybucket_orc;
> Found 99 items
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/00_0
> -rwxrwxrwx 3 hrt_qa hdfs 0 2016-05-23 23:20
> /apps/hive/warehouse/emptybucket_orc/01_0
> ..
> {noformat}
> Expected behavior:
> In tez mode, zero length file shouldn't get created on hdfs if bucket is empty
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14079) Remove file, method and line number from pattern layout
[
https://issues.apache.org/jira/browse/HIVE-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345423#comment-15345423
]
Sergey Shelukhin commented on HIVE-14079:
-
+1 assuming it works :)
> Remove file, method and line number from pattern layout
> ---
>
> Key: HIVE-14079
> URL: https://issues.apache.org/jira/browse/HIVE-14079
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14079.1.patch
>
>
> Using %F%M and %L in pattern layouts need location information which is
> expensive to get and is disabled by default. We should remove them from the
> default layouts. This will avoid creating empty brackets like below
> {code}
> lockmgr.DbTxnManager (:())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
