[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354507#comment-15354507
]
Hive QA commented on HIVE-13945:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814462/HIVE-13945.03.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10288 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/306/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/306/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-306/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814462 - PreCommit-HIVE-MASTER-Build
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: HIVE-13945.03.patch
Changed the default decimal precision in division, not sure why it was so low
by default. May break more tests than it fixes in terms of result changes...
we'll see
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.03.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: HIVE-13945.03.patch
Another round of updates and fixes.
I added double literals and used them in some places where it makes sense.. had
to make some changes to the parser to get around "code too large" errors.
Also adjusted the behavior of some fns to support decimal properly, and fixed
an unrelated bug in Lead/Lag fn exposed by this patch.
Still need to look at precision for operations, and incorrect results for
struct-in (probably related to decimal vs double in the struct), and ORC ppd
test (didn't look yet).
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.03.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353973#comment-15353973
]
Ferdinand Xu commented on HIVE-14037:
-
LGTM +1
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch, HIVE-14037.4.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11653) Beeline asks for password even when connecting with Kerberos
[ https://issues.apache.org/jira/browse/HIVE-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353937#comment-15353937 ] Vihang Karajgaonkar commented on HIVE-11653: HIVE-14062 already fixes this issue. > Beeline asks for password even when connecting with Kerberos > > > Key: HIVE-11653 > URL: https://issues.apache.org/jira/browse/HIVE-11653 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 0.14.0, 1.0.1, 1.2.1, 2.0.0 > Environment: Kerberos and Hive >Reporter: Loïc C. Chanel >Assignee: Oleksiy Sayankin > Attachments: HIVE-11653.1.patch > > > When connecting to HiveServer via Beeline, Beeline asks for a password even > if Kerberos is enabled and there is a ticket in cache (kinit have been > successfully executed, as klist shows the ticket is in cache). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Issue Type: Sub-task (was: Bug) Parent: HIVE-12991 > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[
https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353912#comment-15353912
]
Hive QA commented on HIVE-14035:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814052/HIVE-14035.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 52 failed/errored test(s), 10284 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_acid3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join_part_col_char
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_join_acid_non_acid
org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDump
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptDataFiles
org.apache.hive.hcatalog.streaming.TestStreaming.testFileDumpCorruptSideFiles
org.apache.hive.hcatalog.streaming.TestStreaming.testInterleavedTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testMultipleTransactionBatchCommits
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbortAndCommit
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Delimited
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
{noformat}
Test results:
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: HIVE-14111.01.patch Updated based on comments and added some TODOs > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.patch, > sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353821#comment-15353821 ] Sergey Shelukhin commented on HIVE-14119: - +1 pending tests > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353807#comment-15353807 ] Sergey Shelukhin commented on HIVE-14111: - Wrt close(), I think it should be ok. Right now the session object is often being reused after close; it should work with that. I added another TODO > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Status: Patch Available (was: Open) > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14119) LLAP external recordreader not returning non-ascii string properly
[ https://issues.apache.org/jira/browse/HIVE-14119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14119: -- Attachment: HIVE-14119.1.patch deserializer was created with escapes disabled > LLAP external recordreader not returning non-ascii string properly > -- > > Key: HIVE-14119 > URL: https://issues.apache.org/jira/browse/HIVE-14119 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14119.1.patch > > > Strings with non-ascii chars showing up with "\�\�\� " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[
https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353780#comment-15353780
]
Sergey Shelukhin commented on HIVE-14111:
-
{noformat}
// TODO Lots of exceptions can come out of this one invocation. If not
returned in the finally block - things blow up.
{noformat}
It's inside the try...block. What do you mean?
> better concurrency handling for TezSessionState - part I
>
>
> Key: HIVE-14111
> URL: https://issues.apache.org/jira/browse/HIVE-14111
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14111.patch, sessionPoolNotes.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353787#comment-15353787 ] Sergey Shelukhin commented on HIVE-14111: - What is session.destroy? Rather what object is this; do you mean TezClient::stop? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[
https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353774#comment-15353774
]
Sergey Shelukhin commented on HIVE-14111:
-
{noformat}
// TODO What happens in this case - no new sessions launched ?
{noformat}
Yes.
> better concurrency handling for TezSessionState - part I
>
>
> Key: HIVE-14111
> URL: https://issues.apache.org/jira/browse/HIVE-14111
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14111.patch, sessionPoolNotes.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[
https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353770#comment-15353770
]
Sergey Shelukhin commented on HIVE-14111:
-
{noformat}
if (session == null || conf == null) {
// TODO Will the session being passed in here be null if this is being
used from the set of default sessions.
// Really don't see why - since SessionState can be re-used - in which
case the session will be re-used.
return false;
}
{noformat}
Because it's supposed to be returned to the pool and unset. Same for some other
comments
> better concurrency handling for TezSessionState - part I
>
>
> Key: HIVE-14111
> URL: https://issues.apache.org/jira/browse/HIVE-14111
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14111.patch, sessionPoolNotes.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13884:
---
Attachment: HIVE-13884.9.patch
Here's another patch that includes some tests on TestHiveMetaStore. There was
an issue found by those tests when requesting a MAX number of partitions. It is
fixed now.
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353766#comment-15353766 ] Sergey Shelukhin commented on HIVE-14111: - Actually I think there are many tangentially related TODOs that can be addressed separately. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[
https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353762#comment-15353762
]
Sergey Shelukhin commented on HIVE-14111:
-
{noformat}
// TODO Define a helper class which can accept and validate potential values
HIVE_EXECUTION_MODE("hive.execution.mode", "container", new
StringSet("container", "llap"),
"Chooses whether query fragments will run in container or in llap"),
{noformat}
This is completely unrelated. I'll just ignore other unrelated comments here ;)
> better concurrency handling for TezSessionState - part I
>
>
> Key: HIVE-14111
> URL: https://issues.apache.org/jira/browse/HIVE-14111
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14111.patch, sessionPoolNotes.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14118) Make the alter partition exception more meaningful
[ https://issues.apache.org/jira/browse/HIVE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14118: Status: Patch Available (was: Open) Patch-1: include the underline exception when throwing the exception. > Make the alter partition exception more meaningful > -- > > Key: HIVE-14118 > URL: https://issues.apache.org/jira/browse/HIVE-14118 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14118.1.patch > > > Right now when the alter partitions fails, "alter is not possible" is shown > in the log while the real exception/failure is hidden. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14118) Make the alter partition exception more meaningful
[ https://issues.apache.org/jira/browse/HIVE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14118: Attachment: HIVE-14118.1.patch > Make the alter partition exception more meaningful > -- > > Key: HIVE-14118 > URL: https://issues.apache.org/jira/browse/HIVE-14118 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-14118.1.patch > > > Right now when the alter partitions fails, "alter is not possible" is shown > in the log while the real exception/failure is hidden. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[
https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353738#comment-15353738
]
Hive QA commented on HIVE-14093:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814051/HIVE-14093.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10286 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/292/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/292/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-292/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814051 - PreCommit-HIVE-MASTER-Build
> LLAP output format connection should wait for all writes to finish before
> closing channel
> -
>
> Key: HIVE-14093
> URL: https://issues.apache.org/jira/browse/HIVE-14093
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14093.1.patch, HIVE-14093.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Attachment: HIVE-14027.patch
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Attachment: (was: HIVE-14027.patch)
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Target Version/s: 1.3.0, 2.2.0 (was: 1.3.0, 2.2.0, 2.1.1, 2.0.2)
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14109) query execuction throws NPE when hive.exec.submitviachild is set to true
[
https://issues.apache.org/jira/browse/HIVE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14109:
Status: Patch Available (was: Open)
Patch-1: queryState was null when we call ExecDriver() constructor. Since we
only need QueryId in HadoopJobExecHelper, so pass in QueryId only in this
patch.
> query execuction throws NPE when hive.exec.submitviachild is set to true
>
>
> Key: HIVE-14109
> URL: https://issues.apache.org/jira/browse/HIVE-14109
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14109.1.patch
>
>
> If we set hive.exec.submitviachild to true and execute select count(*) from
> src, the following exception is thrown.
> Seems queryState is not initialized when ExecDriver is called from main() in
> ExecDriver.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14109) query execuction throws NPE when hive.exec.submitviachild is set to true
[
https://issues.apache.org/jira/browse/HIVE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14109:
Attachment: HIVE-14109.1.patch
> query execuction throws NPE when hive.exec.submitviachild is set to true
>
>
> Key: HIVE-14109
> URL: https://issues.apache.org/jira/browse/HIVE-14109
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14109.1.patch
>
>
> If we set hive.exec.submitviachild to true and execute select count(*) from
> src, the following exception is thrown.
> Seems queryState is not initialized when ExecDriver is called from main() in
> ExecDriver.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11986) describe extended command is missing tableType, while describe formatted outputs it
[ https://issues.apache.org/jira/browse/HIVE-11986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353655#comment-15353655 ] Andy Kruth edited comment on HIVE-11986 at 6/28/16 8:31 PM: I see the same behavior. I am trying to find the partitionKey information which comes after the truncation point. I am on: Hive 0.13.0-mapr-1510. You can recreate the issue with this simple table definition: ``` create table simple_table (f1 string) row format delimited fields terminated by '\t'; ``` and see that the describe output is truncated: ``` # col_name data_type comment f1 string # Detailed Table Information Database: default Owner: andy.kruth CreateTime: Tue Jun 28 16:19:04 EDT 2016 LastAccessTime: UNKNOWN Protect Mode: None Retention: 0 Location: maprfs:/user/hive/warehouse/simple_table Table Type: MANAGED_TABLE Table Parameters: transient_lastDdlTime 1467145144 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat:org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets:-1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \t serialization.format\t Time taken: 0.124 seconds, Fetched: 27 row(s) ``` Simply changing the field delimiter specified to something like a comma fixes the issue. Is there a way around this behavior if my table is tab delimited? Addendum: after changing the delimiter to a comma I could see the full describe output including partitionKeys and tableType with 'describe extended'. 'describe formatted' still didn't work as expected. was (Author: kruthar): I see the same behavior. I am trying to find the partitionKey information which comes after the truncation point. I am on: Hive 0.13.0-mapr-1510. You can recreate the issue with this simple table definition: ``` create table simple_table (f1 string) row format delimited fields terminated by '\t'; ``` and see that the describe output is truncated: ``` # col_name data_type comment f1 string # Detailed Table Information Database: default Owner: andy.kruth CreateTime: Tue Jun 28 16:19:04 EDT 2016 LastAccessTime: UNKNOWN Protect Mode: None Retention: 0 Location: maprfs:/user/hive/warehouse/simple_table Table Type: MANAGED_TABLE Table Parameters: transient_lastDdlTime 1467145144 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat:org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets:-1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: field.delim \t serialization.format\t Time taken: 0.124 seconds, Fetched: 27 row(s) ``` Simply changing the field delimiter specified to something like a comma fixes the issue. Is there a way around this behavior if my table is tab delimited? > describe extended command is missing tableType, while describe formatted > outputs it > --- > > Key: HIVE-11986 > URL: https://issues.apache.org/jira/browse/HIVE-11986 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2, Thrift API >Affects Versions: 0.13.0 >Reporter: Aviv >Priority: Minor > > The output of this command "describe extended" is missing the tableType: > property for some tables. it is being truncated after "serialization.format= > , field.delim=" > The output for: describe extended hivesampletable is: > "Detailed Table Information Table(tableName:hivesampletable, > dbName:default, owner:hdpinternaluser, createTime:1440859101, > lastAccessTime:0, retention:0, > sd:StorageDescriptor(cols:[FieldSchema(name:clientid, type:string, > comment:null), FieldSchema(name:querytime, type:string, comment:null), > FieldSchema(name:market, type:string, comment:null), > FieldSchema(name:deviceplatform, type:string, comment:null), > FieldSchema(name:devicemake, type:string, comment:null), > FieldSchema(name:devicemodel, type:string, comment:null), > FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, > type:string, comment:null), FieldSchema(name:querydwelltime, type:double, > comment:null), FieldSchema(name:sessionid, type:bigint,
[jira] [Commented] (HIVE-11986) describe extended command is missing tableType, while describe formatted outputs it
[
https://issues.apache.org/jira/browse/HIVE-11986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353655#comment-15353655
]
Andy Kruth commented on HIVE-11986:
---
I see the same behavior. I am trying to find the partitionKey information which
comes after the truncation point. I am on: Hive 0.13.0-mapr-1510.
You can recreate the issue with this simple table definition:
```
create table simple_table (f1 string)
row format delimited
fields terminated by '\t';
```
and see that the describe output is truncated:
```
# col_name data_type comment
f1 string
# Detailed Table Information
Database: default
Owner: andy.kruth
CreateTime: Tue Jun 28 16:19:04 EDT 2016
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location: maprfs:/user/hive/warehouse/simple_table
Table Type: MANAGED_TABLE
Table Parameters:
transient_lastDdlTime 1467145144
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat:org.apache.hadoop.mapred.TextInputFormat
OutputFormat:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets:-1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
field.delim \t
serialization.format\t
Time taken: 0.124 seconds, Fetched: 27 row(s)
```
Simply changing the field delimiter specified to something like a comma fixes
the issue.
Is there a way around this behavior if my table is tab delimited?
> describe extended command is missing tableType, while describe formatted
> outputs it
> ---
>
> Key: HIVE-11986
> URL: https://issues.apache.org/jira/browse/HIVE-11986
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Thrift API
>Affects Versions: 0.13.0
>Reporter: Aviv
>Priority: Minor
>
> The output of this command "describe extended" is missing the tableType:
> property for some tables. it is being truncated after "serialization.format=
> , field.delim="
> The output for: describe extended hivesampletable is:
> "Detailed Table Information Table(tableName:hivesampletable,
> dbName:default, owner:hdpinternaluser, createTime:1440859101,
> lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:clientid, type:string,
> comment:null), FieldSchema(name:querytime, type:string, comment:null),
> FieldSchema(name:market, type:string, comment:null),
> FieldSchema(name:deviceplatform, type:string, comment:null),
> FieldSchema(name:devicemake, type:string, comment:null),
> FieldSchema(name:devicemodel, type:string, comment:null),
> FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country,
> type:string, comment:null), FieldSchema(name:querydwelltime, type:double,
> comment:null), FieldSchema(name:sessionid, type:bigint, comment:null),
> FieldSchema(name:sessionpagevieworder, type:bigint, comment:null)],
> location:wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format= , field.delim="
> while the output for describe formatted hivesampletable is:
> "# Detailed Table Information
> Database: default
> Owner:hdpinternaluser
> CreateTime: Sat Aug 29 14:38:21 GMT 2015
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location:
> wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable
>
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE true
> numFiles1
> numRows 59793
> rawDataSize 4895922
> totalSize 5015508
> transient_lastDdlTime 1441320153
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353605#comment-15353605 ] Sergey Shelukhin commented on HIVE-14111: - No, all changes are part of this patch... TODO comment removal needs to be done in some patch, that's the only unrelated one > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14091: Attachment: HIVE-14091.01.patch ping [~jdere]. Can you take a look wrt the addition of taskFailed? When running the end to end test, the error is not propagated without it. Sid says it should come via heartbeat. Is heartbeat not coming test specific? > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.01.patch, HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353599#comment-15353599
]
Jesus Camacho Rodriguez edited comment on HIVE-14027 at 6/28/16 7:42 PM:
-
Problem was in the initialization of MapJoin operator. We use the object
inspector from the key of the BigTable in the MapJoin for the key of the
broadcast table, but this is not correct for outer joins (if you have not match
for a left record, the value for the key column in the right side will be
null). I have uploaded a fix and the test case to validate the fix.
was (Author: jcamachorodriguez):
Problem was in the initialization of operator. We use the object inspector from
the key of the BigTable in the MapJoin for the key of the broadcast table, but
this is not correct for outer joins (if you have not match for a left record,
the value for the key column in the right side will be null). I have uploaded a
fix and the test case to validate the fix.
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Attachment: HIVE-14027.patch
Problem was in the initialization of operator. We use the object inspector from
the key of the BigTable in the MapJoin for the key of the broadcast table, but
this is not correct for outer joins (if you have not match for a left record,
the value for the key column in the right side will be null). I have uploaded a
fix and the test case to validate the fix.
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14027.patch
>
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14091) some errors are not propagated to LLAP external clients
[ https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353598#comment-15353598 ] Sergey Shelukhin commented on HIVE-14091: - It's handled via IOException. taskFailed is apparently needed. There isn't a call to the heartbeat with error that I can see; if I remove taskFailed, it never propagates properly. InputStream cleans up the socket (added a comment) > some errors are not propagated to LLAP external clients > --- > > Key: HIVE-14091 > URL: https://issues.apache.org/jira/browse/HIVE-14091 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Sergey Shelukhin > Attachments: HIVE-14091.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Status: Patch Available (was: In Progress)
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-14027 started by Jesus Camacho Rodriguez.
--
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-14027:
--
Assignee: Jesus Camacho Rodriguez (was: Vaibhav Gumashta)
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Jesus Camacho Rodriguez
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13886) HiveSplitGenerator does not read dfs.blocksize correctly
[ https://issues.apache.org/jira/browse/HIVE-13886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353559#comment-15353559 ] shanyu zhao commented on HIVE-13886: The test failure seems irrelevant. > HiveSplitGenerator does not read dfs.blocksize correctly > > > Key: HIVE-13886 > URL: https://issues.apache.org/jira/browse/HIVE-13886 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 1.2.1 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HIVE-13886.patch > > > In src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java, it > uses conf.getLong() to read "dfs.blocksize". This results in Exception if one > configures it to something like "256m". It should use conf.getLongBytes() to > read it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14115) Custom FetchFormatter is not supported
[
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353551#comment-15353551
]
Hive QA commented on HIVE-14115:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814026/HIVE-14115.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10285 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/291/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/291/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-291/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814026 - PreCommit-HIVE-MASTER-Build
> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,
> DefaultFetchFormatter.class.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-13884:
---
Summary: Disallow queries in HMS fetching more than a configured number of
partitions (was: Disallow queries fetching more than a configured number of
partitions in PartitionPruner)
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353450#comment-15353450 ] Szehon Ho commented on HIVE-14117: -- nice idea, +1 > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Attachment: HIVE-14117.1.patch > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-14117: -- Status: Patch Available (was: Open) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353410#comment-15353410
]
Aihua Xu commented on HIVE-14013:
-
The tests are not related. Pushed to master. Thanks Yongzhi for reviewing.
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch, HIVE-14013.4.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14013:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch, HIVE-14013.4.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14037:
Attachment: HIVE-14037.4.patch
Patch-4: address comments.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch, HIVE-14037.4.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353399#comment-15353399
]
Hive QA commented on HIVE-14113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814010/HIVE-14113.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10258 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vectorization_13.q-tez_bmj_schema_evolution.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/289/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/289/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814010 - PreCommit-HIVE-MASTER-Build
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: Navis
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353371#comment-15353371
]
Gunther Hagleitner commented on HIVE-10100:
---
[~sseth] we can ask, but the warning has been around for a long time. My guess
eventually the idea is to get rid of it. So might as well change these places.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353289#comment-15353289
]
Mohit Sabharwal commented on HIVE-13884:
LGTM.
A unit test would be great in TestHiveMetaStore#testListPartitions - we can do
that as a follow-up item.
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353207#comment-15353207
]
Hive QA commented on HIVE-13945:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814004/HIVE-13945.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 10272 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-order_null.q-orc_merge10.q-insert_values_tmp_table.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_lazyserde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_windowing_expressions
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_printf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_example_format
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_struct_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compare_double_bigint
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output
org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput
org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/288/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/288/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 40 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814004 - PreCommit-HIVE-MASTER-Build
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353057#comment-15353057
]
Sergio Peña commented on HIVE-13884:
[~mohitsabharwal] [~szehon] The patch is ready, could you let me know if there
are other comments or if I can commit this?
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-14114: - Assignee: Eugene Koifman (was: Abhishek) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352977#comment-15352977
]
Hive QA commented on HIVE-13901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12814005/HIVE-13901.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10285 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityDelayedAllocation
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/287/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12814005 - PreCommit-HIVE-MASTER-Build
> Hivemetastore add partitions can be slow depending on filesystems
> -
>
> Key: HIVE-13901
> URL: https://issues.apache.org/jira/browse/HIVE-13901
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13901.1.patch, HIVE-13901.2.patch,
> HIVE-13901.6.patch, HIVE-13901.7.patch
>
>
> Depending on FS, creating external tables & adding partitions can be
> expensive (e.g msck which adds all partitions).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14078) LLAP input split should get task attempt number from conf if available
[
https://issues.apache.org/jira/browse/HIVE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352822#comment-15352822
]
Hive QA commented on HIVE-14078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813985/HIVE-14078.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10285 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/286/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/286/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-286/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813985 - PreCommit-HIVE-MASTER-Build
> LLAP input split should get task attempt number from conf if available
> --
>
> Key: HIVE-14078
> URL: https://issues.apache.org/jira/browse/HIVE-14078
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14078.1.patch, HIVE-14078.2.patch
>
>
> Currently the attempt number is hard-coded to 0. If the split is being
> fetched as part of a hadoop job we can get the task attempt ID from the conf
> if it has been set, and use the attempt number from that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[
https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352677#comment-15352677
]
Hive QA commented on HIVE-11402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813993/HIVE-11402.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10285 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813993 - PreCommit-HIVE-MASTER-Build
> HS2 - disallow parallel query execution within a single Session
> ---
>
> Key: HIVE-11402
> URL: https://issues.apache.org/jira/browse/HIVE-11402
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11402.01.patch, HIVE-11402.patch
>
>
> HiveServer2 currently allows concurrent queries to be run in a single
> session. However, every HS2 session has an associated SessionState object,
> and the use of SessionState in many places assumes that only one thread is
> using it, ie it is not thread safe.
> There are many places where SesssionState thread safety needs to be
> addressed, and until then we should serialize all query execution for a
> single HS2 session. -This problem can become more visible with HIVE-4239 now
> allowing parallel query compilation.-
> Note that running queries in parallel for single session is not
> straightforward with jdbc, you need to spawn another thread as the
> Statement.execute calls are blocking. I believe ODBC has non blocking query
> execution API, and Hue is another well known application that shares sessions
> for all queries that a user runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352673#comment-15352673 ] Peter Vary commented on HIVE-14100: --- The jira for the original function > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Priority: Minor > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateAuthenticator the current_user returns the username which > is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352653#comment-15352653
]
Alexandre Linte commented on HIVE-13830:
Hi,
I upgraded Hive to Hive 2.1.0. Now I have the following errors:
- HQL commands
{noformat}
hive (shfs3453)> SELECT COUNT(year) FROM chicagocrimes GROUP BY year;
SELECT COUNT(year) FROM chicagocrimes GROUP BY year
FAILED: SemanticException Failed to get a spark session:
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
{noformat}
- Client logs
{noformat}
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.conf.HiveConfUsing the
default value passed in for log id: c10f51a3-a72d-40c7-9ff6-26e5fb3732da
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.session.SessionStateUpdating thread name to
c10f51a3-a72d-40c7-9ff6-26e5fb3732da main
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.ql.DriverCompiling
command(queryId=shfs3453_20160628110208_f0b51237-d391-472d-abe8-f2dd2457a9ed):
SELECT COUNT(year) FROM chicagocrimes GROUP BY year
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerStarting Semantic Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted phase 1 of Semantic
Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination
tables
Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCompleted getting MetaData in
Semantic Analysis
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for source tables
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for subqueries
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerGet metadata for destination
tables
Jun 28 11:02:08 hive-cli INFO - hive.ql.ContextNew scratch dir is
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO - org.apache.hadoop.hive.common.FileUtilsCreating
directory if it doesn't exist:
hdfs://sandbox/tmp/hive/shfs3453/c10f51a3-a72d-40c7-9ff6-26e5fb3732da/hive_2016-06-28_11-02-08_399_7245611464735028300-1/-mr-10001/.hive-staging_hive_2016-06-28_11-02-08_399_7245611464735028300-1
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.parse.CalcitePlannerCBO Succeeded; optimized logical
plan.
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for FS(6)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(5)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(4)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for RS(3)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for GBY(2)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for SEL(1)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.ppd.OpProcFactoryProcessing for TS(0)
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 oldColExprMap:
{KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactoryRS 3 newColExprMap:
{KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryloading spark
properties from:spark-defaults.conf
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.default.parallelism -> 10).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryoserializer.buffer -> 100m).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.executor.memory -> 4g).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.driver.memory -> 2g).
Jun 28 11:02:08 hive-cli INFO -
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactoryload spark property
from spark-defaults.conf (spark.kryo.classesToRegister ->
[jira] [Updated] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexandre Linte updated HIVE-13830:
---
Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos (was: Hadoop
2.7.2, Hive 2.0.0, Spark 1.6.1, Kerberos)
> Hive on spark driver crash with Spark 1.6.1
> ---
>
> Key: HIVE-13830
> URL: https://issues.apache.org/jira/browse/HIVE-13830
> Project: Hive
> Issue Type: Bug
> Components: Spark, spark-branch
>Affects Versions: 2.0.0, 2.1.0
> Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos
>Reporter: Alexandre Linte
>
> With Hive 1.2.1 I was able to use Hive on Spark successfully with the use of
> the spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar".
> Today with Hive 2.0.0, I'm unable to use Hive on Spark whether it be with the
> spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar" or the spark-assembly
> "spark-assembly-1.6.1-hadoop2.7.2.jar".
> My configuration is the following:
> * spark-default.conf available in HIVE_DIR/conf
> * spark assembly available in HIVE_DIR/lib
> I gathered several logs below:
> - HQL commands
> {noformat}
> $ hive -v --database shfs3453
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Spark/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Logging initialized using configuration in
> file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties
> use shfs3453
> OK
> Time taken: 1.425 seconds
> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
> versions. Consider using a different execution engine (i.e. tez, spark) or
> using Hive 1.X releases.
> hive (shfs3453)> set hive.execution.engine=spark;
> set hive.execution.engine=spark
> hive (shfs3453)> set spark.master=yarn-client;
> set spark.master=yarn-client
> hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING,
> Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING,
> LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT,
> District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT,
> YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude
> FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED
> AS TEXTFILE;
> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block
> STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription
> STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT,
> CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year
> INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE
> OK
> Time taken: 0.408 seconds
> hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM
> chicagocrimes WHERE Description = 'FIRST DEGREE MURDER';
> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE
> Description = 'FIRST DEGREE MURDER'
> Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Starting Spark Job = 79484279-8e75-4b13-8e71-7de463f4d51e
> Status: SENT
> Failed to execute spark task, with exception
> 'java.lang.IllegalStateException(RPC channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> - Client logs
> {noformat}
> May 24 09:32:19 hive-cli WARN -
> org.apache.hive.spark.client.rpc.RpcDispatcherReceived error
> message:io.netty.handler.codec.DecoderException:
> java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job
> at
>
[jira] [Updated] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexandre Linte updated HIVE-13830:
---
Affects Version/s: 2.1.0
> Hive on spark driver crash with Spark 1.6.1
> ---
>
> Key: HIVE-13830
> URL: https://issues.apache.org/jira/browse/HIVE-13830
> Project: Hive
> Issue Type: Bug
> Components: Spark, spark-branch
>Affects Versions: 2.0.0, 2.1.0
> Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos
>Reporter: Alexandre Linte
>
> With Hive 1.2.1 I was able to use Hive on Spark successfully with the use of
> the spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar".
> Today with Hive 2.0.0, I'm unable to use Hive on Spark whether it be with the
> spark-assembly "spark-assembly-1.4.1-hadoop2.7.1.jar" or the spark-assembly
> "spark-assembly-1.6.1-hadoop2.7.2.jar".
> My configuration is the following:
> * spark-default.conf available in HIVE_DIR/conf
> * spark assembly available in HIVE_DIR/lib
> I gathered several logs below:
> - HQL commands
> {noformat}
> $ hive -v --database shfs3453
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Spark/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Logging initialized using configuration in
> file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties
> use shfs3453
> OK
> Time taken: 1.425 seconds
> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
> versions. Consider using a different execution engine (i.e. tez, spark) or
> using Hive 1.X releases.
> hive (shfs3453)> set hive.execution.engine=spark;
> set hive.execution.engine=spark
> hive (shfs3453)> set spark.master=yarn-client;
> set spark.master=yarn-client
> hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING,
> Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING,
> LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT,
> District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT,
> YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude
> FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED
> AS TEXTFILE;
> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block
> STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription
> STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT,
> CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year
> INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE
> OK
> Time taken: 0.408 seconds
> hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM
> chicagocrimes WHERE Description = 'FIRST DEGREE MURDER';
> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE
> Description = 'FIRST DEGREE MURDER'
> Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Starting Spark Job = 79484279-8e75-4b13-8e71-7de463f4d51e
> Status: SENT
> Failed to execute spark task, with exception
> 'java.lang.IllegalStateException(RPC channel is closed.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> - Client logs
> {noformat}
> May 24 09:32:19 hive-cli WARN -
> org.apache.hive.spark.client.rpc.RpcDispatcherReceived error
> message:io.netty.handler.codec.DecoderException:
> java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358)
> at
>
[jira] [Assigned] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek reassigned HIVE-14114: --- Assignee: Abhishek (was: Eugene Koifman) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Abhishek > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek updated HIVE-14114: Status: Open (was: Patch Available) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Abhishek > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[
https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352535#comment-15352535
]
Hive QA commented on HIVE-13369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813981/HIVE-13369.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10273 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/284/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/284/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813981 - PreCommit-HIVE-MASTER-Build
> AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing
> the "best" base file
> --
>
> Key: HIVE-13369
> URL: https://issues.apache.org/jira/browse/HIVE-13369
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Blocker
> Attachments: HIVE-13369.1.patch
>
>
> The JavaDoc on getAcidState() reads, in part:
> "Note that because major compactions don't
>preserve the history, we can't use a base directory that includes a
>transaction id that we must exclude."
> which is correct but there is nothing in the code that does this.
> And if we detect a situation where txn X must be excluded but and there are
> deltas that contain X, we'll have to aborted the txn. This can't
> (reasonably) happen with auto commit mode, but with multi statement txns it's
> possible.
> Suppose some long running txn starts and lock in snapshot at 17 (HWM). An
> hour later it decides to access some partition for which all txns < 20 (for
> example) have already been compacted (i.e. GC'd).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352512#comment-15352512 ] Thejas M Nair commented on HIVE-10815: -- reattaching file to kick off tests before commit > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10815: - Assignee: Nemon Lou (was: Thejas M Nair) > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Nemon Lou > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-10815: - Attachment: HIVE-10815.2.patch > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Thejas M Nair > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10815) Let HiveMetaStoreClient Choose MetaStore Randomly
[ https://issues.apache.org/jira/browse/HIVE-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-10815: Assignee: Thejas M Nair (was: Nemon Lou) > Let HiveMetaStoreClient Choose MetaStore Randomly > - > > Key: HIVE-10815 > URL: https://issues.apache.org/jira/browse/HIVE-10815 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Metastore >Affects Versions: 1.2.0 >Reporter: Nemon Lou >Assignee: Thejas M Nair > Attachments: HIVE-10815.1.patch, HIVE-10815.2.patch, > HIVE-10815.2.patch, HIVE-10815.patch > > > Currently HiveMetaStoreClient using a fixed order to choose MetaStore URIs > when multiple metastores configured. > Choosing MetaStore Randomly will be good for load balance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Issue Type: Improvement (was: Bug) > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: In Progress) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Status: Patch Available (was: Open) > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: In Progress (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14080) hive.metastore.schema.verification should check for schema compatiblity
[ https://issues.apache.org/jira/browse/HIVE-14080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14080: - Attachment: HIVE-14080.1.patch > hive.metastore.schema.verification should check for schema compatiblity > --- > > Key: HIVE-14080 > URL: https://issues.apache.org/jira/browse/HIVE-14080 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14080.1.patch > > > The check done when hive.metastore.schema.verification=true should be based > on compatibility of schema instead of exact version equiality. > See similar change done in schematool - HIVE-12261 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14093: -- Attachment: HIVE-14093.2.patch Add config setting for max pending writes value. > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch, HIVE-14093.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352488#comment-15352488 ] Carl Steinbach commented on HIVE-11402: --- HiveSessionImpl is starting to look a lot like SessionState. I suppose that was inevitable :( > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajat Khandelwal updated HIVE-14113:
Assignee: Navis (was: niklaus xiao)
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: Navis
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14113:
Description:
1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
{quote}
create function my_lower as 'com.tang.UDFLower' using jar
'hdfs:///udf/udf-test.jar';
{quote}
Failed with following exception:
{quote}
0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
INFO : converting to local hdfs:///udf/udf-test.jar
ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
java.lang.RuntimeException: Failed to read external resource
hdfs:///udf/udf-test.jar
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
at
org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
at
org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
at
org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
at
org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs:/udf/udf-test.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
... 28 more
ERROR : Failed to register default.my_lower using class com.tang.UDFLower
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)
{quote}
2. Execute show functions, the failed function my_lower is in the function list
{quote}
0: jdbc:hive2://189.39.151.44:21066/> show functions;
+-+--+
|tab_name |
+-+--+
| day |
| dayofmonth |
| decode |
| default.my_lower |
| degrees |
| dense_rank |
0: jdbc:hive2://189.39.151.44:1/> select my_lower(name) from stu;
Error: Error while compiling statement: FAILED:
