[jira] [Commented] (HIVE-13928) Hive2: float value need to be single quoted inside where clause to return rows when it doesn't have to be
[
https://issues.apache.org/jira/browse/HIVE-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329567#comment-15329567
]
Takahiko Saito commented on HIVE-13928:
---
I don't think anyone is working on it. Cc: [~mmccline] [~jdere]
> Hive2: float value need to be single quoted inside where clause to return
> rows when it doesn't have to be
> -
>
> Key: HIVE-13928
> URL: https://issues.apache.org/jira/browse/HIVE-13928
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Priority: Critical
>
> The below select where with float value does not return any row:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table test;
> No rows affected (0.212 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (f float);
> No rows affected (1.131 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (-35664.76),(29497.34);
> No rows affected (2.482 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.34 |
> ++--+
> 2 rows selected (0.142 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where f =
> -35664.76;
> +-+--+
> | test.f |
> +-+--+
> +-+--+
> {noformat}
> The workaround is to single quote float value:
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where f =
> '-35664.76';
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> ++--+
> 1 row selected (0.163 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14005) Changing a string column of ORC table to boolean sets values to be 'NULL'
[
https://issues.apache.org/jira/browse/HIVE-14005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-14005:
--
Description:
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.015 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.003 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string)
stored as orc;
No rows affected (0.098 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values
('true');
No rows affected (1.091 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| true|
+-+--+
1 row selected (0.114 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s
s boolean;
No rows affected (0.167 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| NULL|
+-+--+
1 row selected (0.098 seconds)
{noformat}
The issue is NOT seen with textfile formatted table:
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.013 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.006 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string)
stored as textfile;
No rows affected (0.116 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values
('true');
No rows affected (1.799 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| true|
+-+--+
1 row selected (0.085 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s
s boolean;
No rows affected (0.141 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| true|
+-+--+
1 row selected (0.093 seconds)
{noformat}
was:
{noformat]
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.015 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.003 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
No rows affected (0.224 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string)
stored as orc;
No rows affected (0.098 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values
('true');
No rows affected (1.091 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| true|
+-+--+
1 row selected (0.114 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column s
s boolean;
No rows affected (0.167 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
+-+--+
| test.s |
+-+--+
| NULL|
+-+--+
1 row selected (0.098 seconds)
{noformat}
The issue is NOT seen with textfile formatted table:
{noformat}
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
hive.metastore.disallow.incompatible.col.type.changes=false;
No rows affected (0.013 seconds)
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
[jira] [Updated] (HIVE-14005) Changing a string column of ORC table to boolean sets values to be 'NULL'
[
https://issues.apache.org/jira/browse/HIVE-14005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-14005:
--
Summary: Changing a string column of ORC table to boolean sets values to be
'NULL' (was: Changing a string column of ORC table to boolean change values to
be 'NULL')
> Changing a string column of ORC table to boolean sets values to be 'NULL'
> -
>
> Key: HIVE-14005
> URL: https://issues.apache.org/jira/browse/HIVE-14005
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.1
>Reporter: Takahiko Saito
>
> {noformat]
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
> hive.metastore.disallow.incompatible.col.type.changes=false;
> No rows affected (0.015 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
> hive.exec.dynamic.partition.mode=nonstrict;
> No rows affected (0.003 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
> No rows affected (0.224 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string)
> stored as orc;
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values
> ('true');
> No rows affected (1.091 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.114 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column
> s s boolean;
> No rows affected (0.167 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s |
> +-+--+
> | NULL|
> +-+--+
> 1 row selected (0.098 seconds)
> {noformat}
> The issue is NOT seen with textfile formatted table:
> {noformat}
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
> hive.metastore.disallow.incompatible.col.type.changes=false;
> No rows affected (0.013 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> set
> hive.exec.dynamic.partition.mode=nonstrict;
> No rows affected (0.006 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> drop table test;
> No rows affected (0.224 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> create table test (s string)
> stored as textfile;
> No rows affected (0.116 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> insert into table test values
> ('true');
> No rows affected (1.799 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.085 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> alter table test change column
> s s boolean;
> No rows affected (0.141 seconds)
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2>
> 0: jdbc:hive2://os-r7-rcmpc-nat-u-hiveserver2> select * from test;
> +-+--+
> | test.s |
> +-+--+
> | true|
> +-+--+
> 1 row selected (0.093 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row
[
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327852#comment-15327852
]
Takahiko Saito edited comment on HIVE-13946 at 6/13/16 6:01 PM:
[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945
The issue here is for Hive 1.
was (Author: taksaito):
[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945
> Decimal value need to be single-quoted when selecting where clause with that
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values
> (4327269606205.029297);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464727816747_0762)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=1, totalSize=21,
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> 4327269606205.029297;
> +--+--+
> | test.dc |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test
> where dc = 4327269606205.029297;
> +--+--+
> | Explain|
> +--+--+
> | STAGE DEPENDENCIES: |
> | Stage-0 is a root stage|
> | |
> | STAGE PLANS: |
> | Stage: Stage-0 |
> | Fetch Operator |
> | limit: -1 |
> | Processor Tree:|
> | TableScan|
> | alias: test|
> | filterExpr: (dc = 4.3272696062050293E12) (type: boolean) |
> | Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean) |
> | Select Operator |
> | expressions: dc (type: decimal(38,18)) |
> | outputColumnNames: _col0 |
> | ListSink |
> |
[jira] [Commented] (HIVE-13946) Decimal value need to be single-quoted when selecting where clause with that decimal value in order to get row
[
https://issues.apache.org/jira/browse/HIVE-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327852#comment-15327852
]
Takahiko Saito commented on HIVE-13946:
---
[~sircodesalot] I believe you are testing with Hive2 and hitting this issue:
https://issues.apache.org/jira/browse/HIVE-13945
> Decimal value need to be single-quoted when selecting where clause with that
> decimal value in order to get row
> --
>
> Key: HIVE-13946
> URL: https://issues.apache.org/jira/browse/HIVE-13946
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
> Fix For: 1.2.1
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value does not return anything.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.175 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (dc
> decimal(38,18));
> No rows affected (0.098 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test values
> (4327269606205.029297);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...327269606205.029297)(Stage-1)
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464727816747_0762)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-04_00-03-54_302_7708281807413586675-940/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=1, totalSize=21,
> rawDataSize=20]
> No rows affected (13.821 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.078 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> 4327269606205.029297;
> +--+--+
> | test.dc |
> +--+--+
> +--+--+
> No rows selected (0.224 seconds)
> {noformat}
> If you single quote that decimal value, a row is returned.
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test where dc =
> '4327269606205.029297';
> +---+--+
> |test.dc|
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.085 seconds)
> {noformat}
> explain shows:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain select * from test
> where dc = 4327269606205.029297;
> +--+--+
> | Explain|
> +--+--+
> | STAGE DEPENDENCIES: |
> | Stage-0 is a root stage|
> | |
> | STAGE PLANS: |
> | Stage: Stage-0 |
> | Fetch Operator |
> | limit: -1 |
> | Processor Tree:|
> | TableScan|
> | alias: test|
> | filterExpr: (dc = 4.3272696062050293E12) (type: boolean) |
> | Filter Operator|
> | predicate: (dc = 4.3272696062050293E12) (type: boolean) |
> | Select Operator |
> | expressions: dc (type: decimal(38,18)) |
> | outputColumnNames: _col0 |
> | ListSink |
> | |
> +--+--+
> 18 rows selected (0.512 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13943) Null is inserted into an existing partition after replacing a column of int with string group type of text formatted partitioned table
[
https://issues.apache.org/jira/browse/HIVE-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13943:
--
Description:
Create a text formatted table with a int column partitioned by a string column.
After replacing the columns of int with string and inserting a new row with the
existing partition ('horton' in this case), null is inserted as a value in the
altered column:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
No rows affected (0.249 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (i int)
partitioned by (s string);
No rows affected (0.116 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test partition
(s) values(1, 'horton');
INFO : Session is already open
INFO : Dag name: insert into table test partition...'horton')(Stage-1)
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1464727816747_0759)
INFO : Map 1: 0/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 1/1
INFO : Loading data to table default.test partition (s=null) from
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-46_619_6747668726633461908-940/-ext-1
INFO : Time taken for load dynamic partitions : 122
INFO : Loading partition {s=horton}
INFO : Time taken for adding to write entity : 0
INFO : Partition default.test{s=horton} stats: [numFiles=1, numRows=1,
totalSize=2, rawDataSize=1]
No rows affected (8.301 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
+-+-+--+
| test.i | test.s |
+-+-+--+
| 1 | horton |
+-+-+--+
1 row selected (0.184 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> alter table test replace columns
(i string);
No rows affected (0.138 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test partition
(s) values ('horton', 'horton');
INFO : Session is already open
INFO : Dag name: insert into table test partition...'horton')(Stage-1)
INFO :
INFO : Map 1: 1/1
INFO : Loading data to table default.test partition (s=null) from
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-55_295_8631413609124947877-940/-ext-1
INFO : Time taken for load dynamic partitions : 190
INFO : Loading partition {s=horton}
INFO : Time taken for adding to write entity : 0
INFO : Partition default.test{s=horton} stats: [numFiles=2, numRows=2,
totalSize=9, rawDataSize=7]
No rows affected (1.35 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
+-+-+--+
| test.i | test.s |
+-+-+--+
| 1 | horton |
| NULL| horton |
+-+-+--+
2 rows selected (0.08 seconds)
{noformat}
The below is explain of insertion:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain insert into table test
partition (s) values ('horton', 'horton');
+---+--+
|Explain
|
+---+--+
| STAGE DEPENDENCIES:
|
| Stage-1 is a root stage
|
| Stage-2 depends on stages: Stage-1
|
| Stage-0 depends on stages: Stage-2
|
| Stage-3 depends on stages: Stage-0
|
|
|
| STAGE PLANS:
|
| Stage: Stage-1
|
| Tez
|
| DagId: hive_20160603211130_4262d739-5bc1-4be0-95c3-2b666f5db7b8:1323
|
| Vertices:
|
| Map 1
[jira] [Updated] (HIVE-13943) Null is inserted into an existing partition after replacing a column of int with string group type of text formatted partitioned table
[
https://issues.apache.org/jira/browse/HIVE-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13943:
--
Summary: Null is inserted into an existing partition after replacing a
column of int with string group type of text formatted partitioned table
(was: Null is inserted into an existing partition after replacing a column of
int with string of text formatted partitioned table )
> Null is inserted into an existing partition after replacing a column of int
> with string group type of text formatted partitioned table
> ---
>
> Key: HIVE-13943
> URL: https://issues.apache.org/jira/browse/HIVE-13943
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Takahiko Saito
>
> Create a text formatted table with a int column partitioned by a string
> column.
> After replacing the columns of int with string and inserting a new row with
> the existing partition ('horton' in this case), null is inserted as a value
> in the altered column:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
> No rows affected (0.249 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (i int)
> partitioned by (s string);
> No rows affected (0.116 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test
> partition (s) values(1, 'horton');
> INFO : Session is already open
> INFO : Dag name: insert into table test partition...'horton')(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464727816747_0759)
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test partition (s=null) from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-46_619_6747668726633461908-940/-ext-1
> INFO :Time taken for load dynamic partitions : 122
> INFO : Loading partition {s=horton}
> INFO :Time taken for adding to write entity : 0
> INFO : Partition default.test{s=horton} stats: [numFiles=1, numRows=1,
> totalSize=2, rawDataSize=1]
> No rows affected (8.301 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +-+-+--+
> | test.i | test.s |
> +-+-+--+
> | 1 | horton |
> +-+-+--+
> 1 row selected (0.184 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> alter table test replace
> columns (i string);
> No rows affected (0.138 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test
> partition (s) values ('horton', 'horton');
> INFO : Session is already open
> INFO : Dag name: insert into table test partition...'horton')(Stage-1)
> INFO :
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test partition (s=null) from
> hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-55_295_8631413609124947877-940/-ext-1
> INFO :Time taken for load dynamic partitions : 190
> INFO : Loading partition {s=horton}
> INFO :Time taken for adding to write entity : 0
> INFO : Partition default.test{s=horton} stats: [numFiles=2, numRows=2,
> totalSize=9, rawDataSize=7]
> No rows affected (1.35 seconds)
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
> +-+-+--+
> | test.i | test.s |
> +-+-+--+
> | 1 | horton |
> | NULL| horton |
> +-+-+--+
> 2 rows selected (0.08 seconds)
> {noformat}
> The below is explain of insertion:
> {noformat}
> 0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain insert into table test
> partition (s) values ('horton', 'horton');
> +---+--+
> |Explain
> |
> +---+--+
> | STAGE DEPENDENCIES:
> |
> | Stage-1 is a root stage
> |
> | Stage-2 depends on stages: Stage-1
>
[jira] [Updated] (HIVE-13943) Null is inserted into an existing partition after replacing a column of int with string of text formatted partitioned table
[
https://issues.apache.org/jira/browse/HIVE-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13943:
--
Description:
Create a text formatted table with a int column partitioned by a string column.
After replacing the columns of int with string and inserting a new row with the
existing partition ('horton' in this case), null is inserted as a value in the
altered column:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> drop table if exists test;
No rows affected (0.249 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> create table test (i int)
partitioned by (s string);
No rows affected (0.116 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test partition
(s) values(1, 'horton');
INFO : Session is already open
INFO : Dag name: insert into table test partition...'horton')(Stage-1)
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1464727816747_0759)
INFO : Map 1: 0/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 0(+1)/1
INFO : Map 1: 1/1
INFO : Loading data to table default.test partition (s=null) from
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-46_619_6747668726633461908-940/-ext-1
INFO : Time taken for load dynamic partitions : 122
INFO : Loading partition {s=horton}
INFO : Time taken for adding to write entity : 0
INFO : Partition default.test{s=horton} stats: [numFiles=1, numRows=1,
totalSize=2, rawDataSize=1]
No rows affected (8.301 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
+-+-+--+
| test.i | test.s |
+-+-+--+
| 1 | horton |
+-+-+--+
1 row selected (0.184 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> alter table test replace columns
(i string);
No rows affected (0.138 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> insert into table test partition
(s) values ('horton', 'horton');
INFO : Session is already open
INFO : Dag name: insert into table test partition...'horton')(Stage-1)
INFO :
INFO : Map 1: 1/1
INFO : Loading data to table default.test partition (s=null) from
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-06-03_21-09-55_295_8631413609124947877-940/-ext-1
INFO : Time taken for load dynamic partitions : 190
INFO : Loading partition {s=horton}
INFO : Time taken for adding to write entity : 0
INFO : Partition default.test{s=horton} stats: [numFiles=2, numRows=2,
totalSize=9, rawDataSize=7]
No rows affected (1.35 seconds)
0: jdbc:hive2://ts-0531-1.openstacklocal:2181>
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> select * from test;
+-+-+--+
| test.i | test.s |
+-+-+--+
| 1 | horton |
| NULL| horton |
+-+-+--+
2 rows selected (0.08 seconds)
{noformat}
The below is explain of insertion:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> explain insert into table test
partition (s) values ('horton', 'horton');
+---+--+
|Explain
|
+---+--+
| STAGE DEPENDENCIES:
|
| Stage-1 is a root stage
|
| Stage-2 depends on stages: Stage-1
|
| Stage-0 depends on stages: Stage-2
|
| Stage-3 depends on stages: Stage-0
|
|
|
| STAGE PLANS:
|
| Stage: Stage-1
|
| Tez
|
| DagId: hive_20160603211130_4262d739-5bc1-4be0-95c3-2b666f5db7b8:1323
|
| Vertices:
|
| Map 1
[jira] [Commented] (HIVE-13935) Changing char column of textfile table to string/varchar leaves white space.
[
https://issues.apache.org/jira/browse/HIVE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314469#comment-15314469
]
Takahiko Saito commented on HIVE-13935:
---
Thanks for your explanation. I have a follow-up question. If non-partitioned
table does not keep original schema, how does conversion of type happen in such
a case? I'm just trying to understand better. Thanks!
> Changing char column of textfile table to string/varchar leaves white space.
>
>
> Key: HIVE-13935
> URL: https://issues.apache.org/jira/browse/HIVE-13935
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> {noformat}
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> create table test_text (c
> char(16)) stored as textfile;
> No rows affected (0.091 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> insert into table test_text
> values ('horton works ');
> INFO : Session is already open
> INFO : Dag name: insert into table test_text values ('ho...')(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464222003837_0399)
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test_text from
> hdfs://os-r6-ifsmes-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test_text/.hive-staging_hive_2016-05-26_17-45-29_669_2888061873550824337-1/-ext-1
> INFO : Table default.test_text stats: [numFiles=1, numRows=1, totalSize=17,
> rawDataSize=16]
> No rows affected (6.849 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.098 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | char(16) | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> alter table test_text change
> column c c string;
> No rows affected (0.145 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | string | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.066 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13935) Changing char column of textfile table to string/varchar leaves white space.
[
https://issues.apache.org/jira/browse/HIVE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313423#comment-15313423
]
Takahiko Saito commented on HIVE-13935:
---
Thanks for the suggestion, [~gopalv]
Yes, HDFS contains extra spaces:
{noformat}
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> dfs -cat
hdfs://ts-0531-5.openstacklocal:8020/apps/hive/warehouse/test_text/00_0
0: jdbc:hive2://ts-0531-1.openstacklocal:2181> ;
+---+--+
|DFS Output |
+---+--+
| horton works |
+---+--+
{noformat}
Cc: [~mmccline]
> Changing char column of textfile table to string/varchar leaves white space.
>
>
> Key: HIVE-13935
> URL: https://issues.apache.org/jira/browse/HIVE-13935
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> {noformat}
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> create table test_text (c
> char(16)) stored as textfile;
> No rows affected (0.091 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> insert into table test_text
> values ('horton works ');
> INFO : Session is already open
> INFO : Dag name: insert into table test_text values ('ho...')(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464222003837_0399)
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test_text from
> hdfs://os-r6-ifsmes-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test_text/.hive-staging_hive_2016-05-26_17-45-29_669_2888061873550824337-1/-ext-1
> INFO : Table default.test_text stats: [numFiles=1, numRows=1, totalSize=17,
> rawDataSize=16]
> No rows affected (6.849 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.098 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | char(16) | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> alter table test_text change
> column c c string;
> No rows affected (0.145 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | string | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.066 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13935) Changing char column of textfile table to string/varchar leaves white space.
[
https://issues.apache.org/jira/browse/HIVE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13935:
--
Summary: Changing char column of textfile table to string/varchar leaves
white space. (was: Changing char column of orc table to string/var char drops
white space.)
> Changing char column of textfile table to string/varchar leaves white space.
>
>
> Key: HIVE-13935
> URL: https://issues.apache.org/jira/browse/HIVE-13935
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> {noformat}
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> create table test_text (c
> char(16)) stored as textfile;
> No rows affected (0.091 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> insert into table test_text
> values ('horton works ');
> INFO : Session is already open
> INFO : Dag name: insert into table test_text values ('ho...')(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464222003837_0399)
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test_text from
> hdfs://os-r6-ifsmes-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test_text/.hive-staging_hive_2016-05-26_17-45-29_669_2888061873550824337-1/-ext-1
> INFO : Table default.test_text stats: [numFiles=1, numRows=1, totalSize=17,
> rawDataSize=16]
> No rows affected (6.849 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.098 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | char(16) | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> alter table test_text change
> column c c string;
> No rows affected (0.145 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> describe test_text;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | c | string | |
> +---++--+--+
> 1 row selected (0.127 seconds)
> 0: jdbc:hive2://os-r6-ifsmes-hiveserver2-11-4> select * from test_text;
> +---+--+
> |test_text.c|
> +---+--+
> | horton works |
> +---+--+
> 1 row selected (0.066 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13851) Change column of textfile formatted table from float to double will drop some fractional digits
[
https://issues.apache.org/jira/browse/HIVE-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13851:
--
Summary: Change column of textfile formatted table from float to double
will drop some fractional digits (was: Change columns of textfile formatted
table from float to double will drop some fractional digits)
> Change column of textfile formatted table from float to double will drop some
> fractional digits
> ---
>
> Key: HIVE-13851
> URL: https://issues.apache.org/jira/browse/HIVE-13851
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> In a case of texfile formatted table, changing a column from float to double
> will drop some fractional digits:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table test(f float);
> No rows affected (0.176 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Tez session hasn't been created yet. Opening session
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: -/-
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-25_17-22-03_941_5459171987449455387-1/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (18.443 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.231 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table test change column
> f f double;
> No rows affected (0.34 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.128 seconds)
> {noformat}
> The issue is not seen with orc table:
> {noformat}
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> create table orc(f float)
> stored as orc;
> No rows affected (0.162 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> insert into table orc
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table orc...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1464135801747_0400)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.orc from
> hdfs://os-r7-wbucju-hiveserver2-11-5.openstacklocal:8020/apps/hive/warehouse/orc/.hive-staging_hive_2016-05-25_17-25-26_569_221649193772012279-1/-ext-1
> INFO : Table default.orc stats: [numFiles=1, numRows=2, totalSize=253,
> rawDataSize=8]
> No rows affected (7.87 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.117 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> alter table orc change column
> f f double;
> No rows affected (0.85 seconds)
> 0: jdbc:hive2://os-r7-wbucju-hiveserver2-11-1> select * from orc;
> +--+--+
> | orc.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.168 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13812) Changing column from float to string group type will drop some factional digits
[
https://issues.apache.org/jira/browse/HIVE-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13812:
--
Summary: Changing column from float to string group type will drop some
factional digits (was: Change column from float to string group type will drop
some factional digits)
> Changing column from float to string group type will drop some factional
> digits
> ---
>
> Key: HIVE-13812
> URL: https://issues.apache.org/jira/browse/HIVE-13812
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> Create a textfile table with a float column and insert some values:
> {noformat}
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> create table test(f float);
> No rows affected (0.237 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> insert into table test
> values(-35664.76171875),(29497.349609375);
> INFO : Session is already open
> INFO : Dag name: insert into table tes...5),(29497.349609375)(Stage-1)
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1463771904371_0006)
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table default.test from
> hdfs://os-r6-pxwhrs-hiveserver2-3re-5.openstacklocal:8020/apps/hive/warehouse/test/.hive-staging_hive_2016-05-20_21-06-29_377_6487823927119226603-10/-ext-1
> INFO : Table default.test stats: [numFiles=1, numRows=2, totalSize=19,
> rawDataSize=17]
> No rows affected (11.069 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
> +--+--+
> | test.f |
> +--+--+
> | -35664.76171875 |
> | 29497.349609375 |
> +--+--+
> 2 rows selected (0.137 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | f | float | |
> +---++--+--+
> 1 row selected (0.173 seconds)
> {noformat}
> Then change float type to string successfully, but when you select table,
> some fractional digits are lost:
> {noformat}
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> alter table test change column
> f f string;
> No rows affected (0.214 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> describe test;
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | f | string | |
> +---++--+--+
> 1 row selected (0.151 seconds)
> 0: jdbc:hive2://os-r6-pxwhrs-hiveserver2-3re-> select * from test;
> ++--+
> | test.f |
> ++--+
> | -35664.76 |
> | 29497.35 |
> ++--+
> 2 rows selected (0.141 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13777) NPE is thrown when select table after change column from string to decimal
[
https://issues.apache.org/jira/browse/HIVE-13777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito resolved HIVE-13777.
---
Resolution: Cannot Reproduce
The issue was not reproduced with later code.
> NPE is thrown when select table after change column from string to decimal
> --
>
> Key: HIVE-13777
> URL: https://issues.apache.org/jira/browse/HIVE-13777
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Run the followings to change a column from string to decimail:
> {noformat}
> drop table if exists
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> create table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> stored as orc as select s, s_dc from schemaevo;
> alter table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> change column s_dc s_dc decimal(12,4);
> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> {noformat}
> select count works fine but just select throws NPE:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> +-+--+
> | c0|
> +-+--+
> | 10 |
> +-+--+
> 1 row selected (13.856 seconds)
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13777) NPE is thrown when select table after change column from string to decimal
[
https://issues.apache.org/jira/browse/HIVE-13777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-13777:
--
Target Version/s: 2.1.0 (was: 1.2.1, 2.1.0)
> NPE is thrown when select table after change column from string to decimal
> --
>
> Key: HIVE-13777
> URL: https://issues.apache.org/jira/browse/HIVE-13777
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>
> Run the followings to change a column from string to decimail:
> {noformat}
> drop table if exists
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> create table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> stored as orc as select s, s_dc from schemaevo;
> alter table
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal
> change column s_dc s_dc decimal(12,4);
> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> {noformat}
> select count works fine but just select throws NPE:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select count(*) from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> +-+--+
> | c0|
> +-+--+
> | 10 |
> +-+--+
> 1 row selected (13.856 seconds)
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from
> shcemaevo_vectorization_true_disallowincompatible_false_fileformat_orc_string_decimal;
> Error: java.io.IOException: java.lang.NullPointerException (state=,code=0)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12810) Hive select fails - java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096700#comment-15096700
]
Takahiko Saito commented on HIVE-12810:
---
If you have hive.server2.enable.doAs=false, you can set it to true, restart hs2
and try again. That may help.
> Hive select fails - java.lang.IndexOutOfBoundsException
> ---
>
> Key: HIVE-12810
> URL: https://issues.apache.org/jira/browse/HIVE-12810
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.2.1
> Environment: HDP 2.3.0
>Reporter: Matjaz Skerjanec
>
> Hadoop HDP 2.3 (Hadoop 2.7.1.2.3.0.0-2557)
> Hive 1.2.1.2.3.0.0-2557
> We are loading orc tables in hive with sqoop from hana db.
> Everything works fine, count and select with ie. 16.000.000 entries in the
> table, but when we load 34.000.000 entries query select does not work anymore
> and we get the followong error (select count(*) is working in both cases):
> {code}
> select count(*) from tablename;
> INFO : Session is already open
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1452091205505_0032)
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: 0/96 Reducer 2: 0/1
> .
> .
> .
> INFO : Map 1: 96/96Reducer 2: 0(+1)/1
> INFO : Map 1: 96/96Reducer 2: 1/1
> +---+--+
> |_c0|
> +---+--+
> | 34146816 |
> +---+--+
> 1 row selected (45.455 seconds)
> {code}
> {code}
> "select originalxml from tablename where messageid =
> 'd0b3c872-435d-499b-a65c-619d9e732bbb'
> 0: jdbc:hive2://10.4.zz.xx:1/default> select originalxml from tablename
> where messageid = 'd0b3c872-435d-499b-a65c-619d9e732bbb';
> INFO : Session is already open
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1452091205505_0032)
> INFO : Map 1: -/-
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1452091205505_0032_1_00, diagnostics=[Vertex
> vertex_1452091205505_0032_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tablename initializer failed,
> vertex=vertex_1452091205505_0032_1_00 [Map 1], java.lang.RuntimeException:
> serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1016)
> ... 15 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0
> at java.util.Collections$EmptyList.get(Collections.java:4454)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Type.getSubtypes(OrcProto.java:12240)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getColumnIndicesFromNames(ReaderImpl.java:649)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getRawDataSizeOfColumns(ReaderImpl.java:632)
> at
>
[jira] [Updated] (HIVE-12778) Having with count distinct doesn't work for special combination
[
https://issues.apache.org/jira/browse/HIVE-12778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12778:
--
Affects Version/s: 1.2.1
> Having with count distinct doesn't work for special combination
> ---
>
> Key: HIVE-12778
> URL: https://issues.apache.org/jira/browse/HIVE-12778
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0, 1.2.1
>Reporter: Peter Brejcak
>
> There is problem for combination of count(distinct ) in having clause without
> count(distinct ) in select clause.
> First case returns error *FAILED: SemanticException [Error 10002]: Line
> Invalid column reference* (unexpected)
> If I add count(distinct ) to select clause result is ok (expected).
> Please run code to see it.
> Steps to reproduce:
> {code}
> create table table_subquery_having_problem (id int, value int);
> insert into table table_subquery_having_problem values (1,1);
> insert into table table_subquery_having_problem values (1,2);
> insert into table table_subquery_having_problem values (1,3);
> insert into table table_subquery_having_problem values (1,4);
> insert into table table_subquery_having_problem values (1,5);
> insert into table table_subquery_having_problem values (1,6);
> insert into table table_subquery_having_problem values (1,7);
> insert into table table_subquery_having_problem values (1,8);
> insert into table table_subquery_having_problem values (1,9);
> select x.id from table_subquery_having_problem x
> group by x.id
> having count(distinct x.value)>1; -- result is ERROR
> select x.id, count(distinct x.value) from table_subquery_having_problem x
> group by x.id
> having count(distinct x.value)>1; --result is OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12778) Having with count distinct doesn't work for special combination
[
https://issues.apache.org/jira/browse/HIVE-12778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095390#comment-15095390
]
Takahiko Saito commented on HIVE-12778:
---
I was able to reproduce with ver. 1.2.1 with the below stack trace:
{noformat}
FAILED: SemanticException [Error 10002]: Line 3:22 Invalid column reference
'value'
16/01/13 01:16:06 [main]: ERROR ql.Driver: FAILED: SemanticException [Error
10002]: Line 3:22 Invalid column reference 'value'
org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:22 Invalid column
reference 'value'
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanGroupByOperator1(SemanticAnalyzer.java:4492)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5775)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8883)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9743)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9636)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10109)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:329)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10120)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:211)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:454)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:314)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1164)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1212)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}
> Having with count distinct doesn't work for special combination
> ---
>
> Key: HIVE-12778
> URL: https://issues.apache.org/jira/browse/HIVE-12778
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.1.0, 1.2.1
>Reporter: Peter Brejcak
>
> There is problem for combination of count(distinct ) in having clause without
> count(distinct ) in select clause.
> First case returns error *FAILED: SemanticException [Error 10002]: Line
> Invalid column reference* (unexpected)
> If I add count(distinct ) to select clause result is ok (expected).
> Please run code to see it.
> Steps to reproduce:
> {code}
> create table table_subquery_having_problem (id int, value int);
> insert into table table_subquery_having_problem values (1,1);
> insert into table table_subquery_having_problem values (1,2);
> insert into table table_subquery_having_problem values (1,3);
> insert into table table_subquery_having_problem values (1,4);
> insert into table table_subquery_having_problem values (1,5);
> insert into table table_subquery_having_problem values (1,6);
> insert into table table_subquery_having_problem values (1,7);
> insert into table table_subquery_having_problem values (1,8);
> insert into table table_subquery_having_problem values (1,9);
> select x.id from table_subquery_having_problem x
> group by x.id
> having count(distinct x.value)>1; -- result is ERROR
> select x.id, count(distinct x.value) from table_subquery_having_problem x
> group by x.id
> having count(distinct x.value)>1; --result is OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12810) Hive select fails - java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095298#comment-15095298
]
Takahiko Saito commented on HIVE-12810:
---
[~maske], Have you checked the hiveserver2 log to see if hiveserver2 has
started properly? Do you seen any error there?
> Hive select fails - java.lang.IndexOutOfBoundsException
> ---
>
> Key: HIVE-12810
> URL: https://issues.apache.org/jira/browse/HIVE-12810
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Affects Versions: 1.2.1
> Environment: HDP 2.3.0
>Reporter: Matjaz Skerjanec
>
> Hadoop HDP 2.3 (Hadoop 2.7.1.2.3.0.0-2557)
> Hive 1.2.1.2.3.0.0-2557
> We are loading orc tables in hive with sqoop from hana db.
> Everything works fine, count and select with ie. 16.000.000 entries in the
> table, but when we load 34.000.000 entries query select does not work anymore
> and we get the followong error (select count(*) is working in both cases):
> {code}
> select count(*) from tablename;
> INFO : Session is already open
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1452091205505_0032)
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: 0/96 Reducer 2: 0/1
> .
> .
> .
> INFO : Map 1: 96/96Reducer 2: 0(+1)/1
> INFO : Map 1: 96/96Reducer 2: 1/1
> +---+--+
> |_c0|
> +---+--+
> | 34146816 |
> +---+--+
> 1 row selected (45.455 seconds)
> {code}
> {code}
> "select originalxml from tablename where messageid =
> 'd0b3c872-435d-499b-a65c-619d9e732bbb'
> 0: jdbc:hive2://10.4.zz.xx:1/default> select originalxml from tablename
> where messageid = 'd0b3c872-435d-499b-a65c-619d9e732bbb';
> INFO : Session is already open
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1452091205505_0032)
> INFO : Map 1: -/-
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1452091205505_0032_1_00, diagnostics=[Vertex
> vertex_1452091205505_0032_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tablename initializer failed,
> vertex=vertex_1452091205505_0032_1_00 [Map 1], java.lang.RuntimeException:
> serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1021)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1048)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 0
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1016)
> ... 15 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0
> at java.util.Collections$EmptyList.get(Collections.java:4454)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcProto$Type.getSubtypes(OrcProto.java:12240)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getColumnIndicesFromNames(ReaderImpl.java:649)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.getRawDataSizeOfColumns(ReaderImpl.java:632)
> at
>
[jira] [Resolved] (HIVE-11870) When exporting an external table, importing it, and dropping it, hdfs data is not removed.
[
https://issues.apache.org/jira/browse/HIVE-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito resolved HIVE-11870.
---
Resolution: Won't Fix
This was expected because table property for external table is true.
> When exporting an external table, importing it, and dropping it, hdfs data is
> not removed.
> ---
>
> Key: HIVE-11870
> URL: https://issues.apache.org/jira/browse/HIVE-11870
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> Export a table to some dir and import it with a different table name.
> {noformat}
> export table student to '/tmp/student';
> import table imported_student from '/tmp/student';
> {noformat}
> Then drop the imported table successfully
> {noformat}
> drop table imported_student;
> {noformat}
> but hdfs for that table still exists
> {noformat}
> hadoop fs -ls /apps/hive/warehouse/imported_student
> Found 1 items
> -rwxrwxrwx 3 hrt_qa hdfs 219190 2015-09-17 22:19
> /apps/hive/warehouse/imported_student/studenttab10k
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and vectorization is enabled.
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12435:
--
Summary: SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case
of ORC and vectorization is enabled. (was: SELECT COUNT(CASE WHEN...) GROUPBY
returns 1 for 'NULL' in a case of ORC)
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and
> vectorization is enabled.
> --
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12435:
--
Summary: SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case
of ORC (was: SELECT COUNT(CASE WHEN...) GROUPBY returns wrong results in a
case of ORC)
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC
>
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and vectorization is enabled.
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12435:
--
Description:
Run the following query:
{noformat}
create table count_case_groupby (key string, bool boolean) STORED AS orc;
insert into table count_case_groupby values ('key1', true),('key2',
false),('key3', NULL),('key4', false),('key5',NULL);
{noformat}
The table contains the following:
{noformat}
key1true
key2false
key3NULL
key4false
key5NULL
{noformat}
The below query returns:
{noformat}
SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END) AS
cnt_bool0_ok FROM count_case_groupby GROUP BY key;
key11
key21
key31
key41
key51
{noformat}
while it expects the following results:
{noformat}
key11
key21
key30
key41
key50
{noformat}
The query works with hive ver 1.2. Also it works when a table is not orc format.
Also even if it's an orc table, when vectorization is disabled, the query works.
was:
Run the following query:
{noformat}
create table count_case_groupby (key string, bool boolean) STORED AS orc;
insert into table count_case_groupby values ('key1', true),('key2',
false),('key3', NULL),('key4', false),('key5',NULL);
{noformat}
The table contains the following:
{noformat}
key1true
key2false
key3NULL
key4false
key5NULL
{noformat}
The below query returns:
{noformat}
SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END) AS
cnt_bool0_ok FROM count_case_groupby GROUP BY key;
key11
key21
key31
key41
key51
{noformat}
while it expects the following results:
{noformat}
key11
key21
key30
key41
key50
{noformat}
The query works with hive ver 1.2. Also it works when a table is not orc format.
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and
> vectorization is enabled.
> --
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
> Also even if it's an orc table, when vectorization is disabled, the query
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12435) SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and vectorization is enabled.
[
https://issues.apache.org/jira/browse/HIVE-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009292#comment-15009292
]
Takahiko Saito commented on HIVE-12435:
---
Thanks for the info, [~gopalv]. That explains why the above passes with hive v
1.2.
There is also another query which fails:
I have the table:
{noformat}
select bool0, key, num0, num1 from calcs;
NULLkey09 NULL12.4
truekey10 NULL10.32
false key11 NULL2.47
NULLkey12 NULL12.05
NULLkey13 NULL10.37
truekey14 NULL7.1
false key15 NULL16.81
NULLkey16 NULL7.12
truekey00 12.38.42
false key01 -12.3 6.71
NULLkey02 15.79.78
truekey03 -15.7 7.43
false key04 3.5 9.05
NULLkey05 -3.59.38
truekey06 0.0 16.42
false key07 NULL11.38
NULLkey08 10.09.47
{noformat}
When I run the following with vectorization enabled, it returns the following:
{noformat}
SELECT bool0, key, SUM((CASE WHEN bool0 THEN CAST(num0 AS DOUBLE) WHEN NOT
bool0 THEN CAST(num1 AS DOUBLE) ELSE NULL END)) FROM Calcs GROUP BY bool0, key,
num1;
NULL key02 0.0
NULL key05 0.0
NULL key08 0.0
NULL key09 0.0
NULL key12 0.0
NULL key13 0.0
NULL key16 0.0
falsekey01 6.71
falsekey04 9.05
falsekey07 11.38
falsekey11 2.47
falsekey15 16.81
truekey00 12.3
truekey03 -15.7
truekey06 0.0
truekey10 NULL
truekey14 NULL
{noformat}
When vectorization is disabled, it returns NULL for the sum instead of 0 when
bool0=NULL :
{noformat}
NULL key02 NULL
NULL key05 NULL
NULL key08 NULL
NULL key09 NULL
NULL key12 NULL
NULL key13 NULL
NULL key16 NULL
falsekey01 6.71
falsekey04 9.05
falsekey07 11.38
falsekey11 2.47
falsekey15 16.81
true key00 12.3
true key03 -15.7
true key06 0.0
true key10 NULL
true key14 NULL
{noformat}
Please let me know in a case if this needs to be addressed in another JIRA.
> SELECT COUNT(CASE WHEN...) GROUPBY returns 1 for 'NULL' in a case of ORC and
> vectorization is enabled.
> --
>
> Key: HIVE-12435
> URL: https://issues.apache.org/jira/browse/HIVE-12435
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Assignee: Gopal V
>
> Run the following query:
> {noformat}
> create table count_case_groupby (key string, bool boolean) STORED AS orc;
> insert into table count_case_groupby values ('key1', true),('key2',
> false),('key3', NULL),('key4', false),('key5',NULL);
> {noformat}
> The table contains the following:
> {noformat}
> key1 true
> key2 false
> key3 NULL
> key4 false
> key5 NULL
> {noformat}
> The below query returns:
> {noformat}
> SELECT key, COUNT(CASE WHEN bool THEN 1 WHEN NOT bool THEN 0 ELSE NULL END)
> AS cnt_bool0_ok FROM count_case_groupby GROUP BY key;
> key1 1
> key2 1
> key3 1
> key4 1
> key5 1
> {noformat}
> while it expects the following results:
> {noformat}
> key1 1
> key2 1
> key3 0
> key4 1
> key5 0
> {noformat}
> The query works with hive ver 1.2. Also it works when a table is not orc
> format.
> Also even if it's an orc table, when vectorization is disabled, the query
> works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12420) WebHCat server throws NPE when you run command with -d user.name.
[
https://issues.apache.org/jira/browse/HIVE-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-12420:
--
Summary: WebHCat server throws NPE when you run command with -d user.name.
(was: Uploaded image for project: 'Bug DB' Bug DBBUG-48161 WebHCat server
throws NPE when you run command with -d user.name.)
> WebHCat server throws NPE when you run command with -d user.name.
> -
>
> Key: HIVE-12420
> URL: https://issues.apache.org/jira/browse/HIVE-12420
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 2.0.0
>Reporter: Takahiko Saito
>Assignee: Eugene Koifman
>
> When you run with '-d user.name', it failed with:
> {noformat}
> [hrt_qa@os-r6-bccslu-hive-1-r-5 ~]$ curl -s -d user.name=hrt_qa -d
> execute="drop table if exists templetontest_tab2;"
> http://os-r6-bccslu-hive-1-r-3.novalocal:20111/templeton/v1/ddl
>
>
>
> Error 500 Server Error
>
>
> HTTP ERROR: 500
> Problem accessing /templeton/v1/ddl. Reason:
> Server Error
> Powered by Jetty://
>
>
> {noformat}
> server log shows:
> {noformat}
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
> java.lang.NullPointerException
> at
> org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:235)
> ~[hive-jdbc-1.2.1.2.3.5.0-13-standalone.jar:1.2.1.2.3.5.0-13]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:143)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:179)
> ~[hadoop-auth-2.6.0.jar:?]
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
> ~[hadoop-auth-2.6.0.jar:?]
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:88)
> ~[hadoop-hdfs-2.7.1.2.3.5.0-13.jar:?]
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> ~[jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> [jetty-all-7.6.0.v20120127.jar:7.6.0.v20120127]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> WARN | 16 Nov 2015 19:48:22,738 | org.eclipse.jetty.servlet.ServletHandler |
> /templeton/v1/ddl
>
[jira] [Updated] (HIVE-11870) When exporting an external table, importing it, and dropping it, hdfs data is not removed.
[
https://issues.apache.org/jira/browse/HIVE-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11870:
--
Summary: When exporting an external table, importing it, and dropping it,
hdfs data is not removed. (was: When dropping an imported table, hdfs for
that table is not removed. )
> When exporting an external table, importing it, and dropping it, hdfs data is
> not removed.
> ---
>
> Key: HIVE-11870
> URL: https://issues.apache.org/jira/browse/HIVE-11870
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>
> Export a table to some dir and import it with a different table name.
> {noformat}
> export table student to '/tmp/student';
> import table imported_student from '/tmp/student';
> {noformat}
> Then drop the imported table successfully
> {noformat}
> drop table imported_student;
> {noformat}
> but hdfs for that table still exists
> {noformat}
> hadoop fs -ls /apps/hive/warehouse/imported_student
> Found 1 items
> -rwxrwxrwx 3 hrt_qa hdfs 219190 2015-09-17 22:19
> /apps/hive/warehouse/imported_student/studenttab10k
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5854) query fails in vectorized mode on empty partitioned table
[ https://issues.apache.org/jira/browse/HIVE-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983590#comment-14983590 ] Takahiko Saito commented on HIVE-5854: -- The issue is no longer seen in hive-0.14. It might have been fixed by HIVE-10273. > query fails in vectorized mode on empty partitioned table > - > > Key: HIVE-5854 > URL: https://issues.apache.org/jira/browse/HIVE-5854 > Project: Hive > Issue Type: Sub-task >Reporter: Eric Hanson > > create table customer ( > c_customer_sk int, > c_customer_id string, > c_current_cdemo_sk int, > c_current_hdemo_sk int, > c_current_addr_sk int, > c_first_shipto_date_sk int, > c_first_sales_date_sk int, > c_salutation string, > c_first_name string, > c_last_name string, > c_preferred_cust_flag string, > c_birth_day int, > c_birth_month int, > c_birth_country string, > c_login string, > c_email_address string, > c_last_review_date string > ) > partitioned by(c_birth_year int) > stored as orc; > set hive.vectorized.execution.enabled = true; > -- fails > select CAST(CAST(c_customer_sk as float) as string),c_first_name,c_last_name > from customer; > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.ql.exec.OperatorFactory.getVectorOperator(OperatorFactory.java:137) > at > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeOperator(Vectorizer.java:640) > at > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationNodeProcessor.process(Vectorizer.java:384) > at > org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) > at > org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:54) > at > org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:59) > at > org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) > at > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.vectorizeMRTask(Vectorizer.java:278) > at > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:219) > at > org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111) > at > org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194) > at > org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139) > at > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.resolve(Vectorizer.java:433) > at > org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100) > at > org.apache.hadoop.hive.ql.parse.MapReduceCompiler.compile(MapReduceCompiler.java:300) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8884) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:292) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:341) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:994) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:905) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.NullPointerException >
[jira] [Commented] (HIVE-9050) NULL values for empty strings when joining with ORC table
[ https://issues.apache.org/jira/browse/HIVE-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983555#comment-14983555 ] Takahiko Saito commented on HIVE-9050: -- I tried with hive-0.14.0 and the issue was not reproduced, either. > NULL values for empty strings when joining with ORC table > - > > Key: HIVE-9050 > URL: https://issues.apache.org/jira/browse/HIVE-9050 > Project: Hive > Issue Type: Bug > Components: SQL >Affects Versions: 0.13.0 > Environment: CentOS release 6.4 (Final), Hortonworks 2.1, Tez > Hive 0.13.0.2.1.3.0-563 > Subversion > git://ip-10-0-0-91/grid/0/jenkins/workspace/BIGTOP-HDP_RPM_REPO-baikal-GA-centos6/bigtop/build/hive/rpm/BUILD/h > ive-0.13.0.2.1.3.0 -r a738a76c72d6d9dd304691faada57a94429256bc > Compiled by jenkins on Thu Jun 26 18:28:50 EDT 2014 > From source with checksum 4dbd99dd254f0c521ad8ab072045325d >Reporter: Usein Faradzhev > > When ORC table contains an empty strings and the SQL query contains at least > one join a hive returns NULL instead of empty values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11603) IndexOutOfBoundsException thrown when accessing a union all subquery and filtering on a column which does not exist in all underlying tables
[
https://issues.apache.org/jira/browse/HIVE-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11603:
--
Affects Version/s: 0.14.0
1.2.1
> IndexOutOfBoundsException thrown when accessing a union all subquery and
> filtering on a column which does not exist in all underlying tables
>
>
> Key: HIVE-11603
> URL: https://issues.apache.org/jira/browse/HIVE-11603
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0, 1.3.0, 1.2.1
> Environment: Hadoop 2.6
>Reporter: Nicholas Brenwald
>Priority: Minor
> Fix For: 2.0.0
>
>
> Create two empty tables t1 and t2
> {code}
> CREATE TABLE t1(c1 STRING);
> CREATE TABLE t2(c1 STRING, c2 INT);
> {code}
> Create a view on these two tables
> {code}
> CREATE VIEW v1 AS
> SELECT c1, c2
> FROM (
> SELECT c1, CAST(NULL AS INT) AS c2 FROM t1
> UNION ALL
> SELECT c1, c2 FROM t2
> ) x;
> {code}
> Then run
> {code}
> SELECT COUNT(*) from v1
> WHERE c2 = 0;
> {code}
> We expect to get a result of zero, but instead the query fails with stack
> trace:
> {code}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:119)
> ... 22 more
> {code}
> Workarounds include disabling ppd,
> {code}
> set hive.optimize.ppd=false;
> {code}
> Or changing the view so that column c2 is null cast to double:
> {code}
> CREATE VIEW v1_workaround AS
> SELECT c1, c2
> FROM (
> SELECT c1, CAST(NULL AS DOUBLE) AS c2 FROM t1
> UNION ALL
> SELECT c1, c2 FROM t2
> ) x;
> {code}
> The problem seems to occur in branch-1.1, branch-1.2, branch-1 but seems to
> be resolved in master (2.0.0)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11603) IndexOutOfBoundsException thrown when accessing a union all subquery and filtering on a column which does not exist in all underlying tables
[
https://issues.apache.org/jira/browse/HIVE-11603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980845#comment-14980845
]
Takahiko Saito commented on HIVE-11603:
---
I tried with hive ver 0.14 and 1.2 and the issue exists. I will add those two
versions to affects version.
Also the issue does not exist if hive.execution.engine=tez.
> IndexOutOfBoundsException thrown when accessing a union all subquery and
> filtering on a column which does not exist in all underlying tables
>
>
> Key: HIVE-11603
> URL: https://issues.apache.org/jira/browse/HIVE-11603
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0
> Environment: Hadoop 2.6
>Reporter: Nicholas Brenwald
>Priority: Minor
> Fix For: 2.0.0
>
>
> Create two empty tables t1 and t2
> {code}
> CREATE TABLE t1(c1 STRING);
> CREATE TABLE t2(c1 STRING, c2 INT);
> {code}
> Create a view on these two tables
> {code}
> CREATE VIEW v1 AS
> SELECT c1, c2
> FROM (
> SELECT c1, CAST(NULL AS INT) AS c2 FROM t1
> UNION ALL
> SELECT c1, c2 FROM t2
> ) x;
> {code}
> Then run
> {code}
> SELECT COUNT(*) from v1
> WHERE c2 = 0;
> {code}
> We expect to get a result of zero, but instead the query fails with stack
> trace:
> {code}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:442)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:119)
> ... 22 more
> {code}
> Workarounds include disabling ppd,
> {code}
> set hive.optimize.ppd=false;
> {code}
> Or changing the view so that column c2 is null cast to double:
> {code}
> CREATE VIEW v1_workaround AS
> SELECT c1, c2
> FROM (
> SELECT c1, CAST(NULL AS DOUBLE) AS c2 FROM t1
> UNION ALL
> SELECT c1, c2 FROM t2
> ) x;
> {code}
> The problem seems to occur in branch-1.1, branch-1.2, branch-1 but seems to
> be resolved in master (2.0.0)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11820) export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"
[
https://issues.apache.org/jira/browse/HIVE-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14899955#comment-14899955
]
Takahiko Saito commented on HIVE-11820:
---
Thank you, [~ashutoshc]!
> export tables with size of >32MB throws "java.lang.IllegalArgumentException:
> Skip CRC is valid only with update options"
>
>
> Key: HIVE-11820
> URL: https://issues.apache.org/jira/browse/HIVE-11820
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Takahiko Saito
>Assignee: Takahiko Saito
> Fix For: 2.0.0
>
> Attachments: HIVE-11820.2.patch, HIVE-11820.2.patch, HIVE-11820.patch
>
>
> Tested a patch of HIVE-11607 and seeing the following exception:
> {noformat}
> 2015-09-14 21:44:16,817 ERROR [main]: exec.Task
> (SessionState.java:printError(960)) - Failed with exception Skip CRC is valid
> only with update options
> java.lang.IllegalArgumentException: Skip CRC is valid only with update options
> at
> org.apache.hadoop.tools.DistCpOptions.validate(DistCpOptions.java:556)
> at
> org.apache.hadoop.tools.DistCpOptions.setSkipCRC(DistCpOptions.java:311)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1147)
> at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553)
> at org.apache.hadoop.hive.ql.exec.CopyTask.execute(CopyTask.java:82)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> A possible resolution is to reverse the order of the following two lines from
> a patch of HIVE-11607:
> {noformat}
> +options.setSkipCRC(true);
> +options.setSyncFolder(true);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11820) export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"
[
https://issues.apache.org/jira/browse/HIVE-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11820:
--
Attachment: HIVE-11820.patch
> export tables with size of >32MB throws "java.lang.IllegalArgumentException:
> Skip CRC is valid only with update options"
>
>
> Key: HIVE-11820
> URL: https://issues.apache.org/jira/browse/HIVE-11820
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Takahiko Saito
>Assignee: Takahiko Saito
> Fix For: 1.2.1
>
> Attachments: HIVE-11820.patch
>
>
> Tested a patch of HIVE-11607 and seeing the following exception:
> {noformat}
> 2015-09-14 21:44:16,817 ERROR [main]: exec.Task
> (SessionState.java:printError(960)) - Failed with exception Skip CRC is valid
> only with update options
> java.lang.IllegalArgumentException: Skip CRC is valid only with update options
> at
> org.apache.hadoop.tools.DistCpOptions.validate(DistCpOptions.java:556)
> at
> org.apache.hadoop.tools.DistCpOptions.setSkipCRC(DistCpOptions.java:311)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1147)
> at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553)
> at org.apache.hadoop.hive.ql.exec.CopyTask.execute(CopyTask.java:82)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> A possible resolution is to reverse the order of the following two lines from
> a patch of HIVE-11607:
> {noformat}
> +options.setSkipCRC(true);
> +options.setSyncFolder(true);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11820) export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"
[
https://issues.apache.org/jira/browse/HIVE-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11820:
--
Fix Version/s: (was: 1.2.1)
> export tables with size of >32MB throws "java.lang.IllegalArgumentException:
> Skip CRC is valid only with update options"
>
>
> Key: HIVE-11820
> URL: https://issues.apache.org/jira/browse/HIVE-11820
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Takahiko Saito
>Assignee: Takahiko Saito
> Attachments: HIVE-11820.patch
>
>
> Tested a patch of HIVE-11607 and seeing the following exception:
> {noformat}
> 2015-09-14 21:44:16,817 ERROR [main]: exec.Task
> (SessionState.java:printError(960)) - Failed with exception Skip CRC is valid
> only with update options
> java.lang.IllegalArgumentException: Skip CRC is valid only with update options
> at
> org.apache.hadoop.tools.DistCpOptions.validate(DistCpOptions.java:556)
> at
> org.apache.hadoop.tools.DistCpOptions.setSkipCRC(DistCpOptions.java:311)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1147)
> at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553)
> at org.apache.hadoop.hive.ql.exec.CopyTask.execute(CopyTask.java:82)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> A possible resolution is to reverse the order of the following two lines from
> a patch of HIVE-11607:
> {noformat}
> +options.setSkipCRC(true);
> +options.setSyncFolder(true);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11820) export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"
[
https://issues.apache.org/jira/browse/HIVE-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11820:
--
Affects Version/s: (was: 1.2.1)
> export tables with size of >32MB throws "java.lang.IllegalArgumentException:
> Skip CRC is valid only with update options"
>
>
> Key: HIVE-11820
> URL: https://issues.apache.org/jira/browse/HIVE-11820
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Takahiko Saito
>Assignee: Takahiko Saito
> Attachments: HIVE-11820.patch
>
>
> Tested a patch of HIVE-11607 and seeing the following exception:
> {noformat}
> 2015-09-14 21:44:16,817 ERROR [main]: exec.Task
> (SessionState.java:printError(960)) - Failed with exception Skip CRC is valid
> only with update options
> java.lang.IllegalArgumentException: Skip CRC is valid only with update options
> at
> org.apache.hadoop.tools.DistCpOptions.validate(DistCpOptions.java:556)
> at
> org.apache.hadoop.tools.DistCpOptions.setSkipCRC(DistCpOptions.java:311)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1147)
> at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553)
> at org.apache.hadoop.hive.ql.exec.CopyTask.execute(CopyTask.java:82)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> A possible resolution is to reverse the order of the following two lines from
> a patch of HIVE-11607:
> {noformat}
> +options.setSkipCRC(true);
> +options.setSyncFolder(true);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11589) Invalid value such as '-1' should be checked for 'hive.txn.timeout'.
[
https://issues.apache.org/jira/browse/HIVE-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-11589:
--
Description:
When an user accidentally set an invalid value such as '-1' for
'hive.txn.timeout', the query simply fails throwing 'NoSuchLockException'
{noformat}
2015-08-16 23:25:43,149 ERROR [HiveServer2-Background-Pool: Thread-206]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
NoSuchLockException(message:No such lock: 40)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatLock(TxnHandler.java:1710)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.unlock(TxnHandler.java:501)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.unlock(HiveMetaStore.java:5571)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy7.unlock(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.unlock(HiveMetaStoreClient.java:1876)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy8.unlock(Unknown Source)
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.unlock(DbLockManager.java:134)
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.releaseLocks(DbLockManager.java:153)
at
org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1038)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1208)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The better way to handle such an invalid value is to check the value beforehand
instead of throwing NoSuchLockException.
was:
When an user accidentally set an invalid value such as '-1' for
'hive.txn.timeout', the query simply fails throwing 'NoSuchLockException'
{noformat}
2015-08-16 23:25:43,149 ERROR [HiveServer2-Background-Pool: Thread-206]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
NoSuchLockException(message:No such lock: 40)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.heartbeatLock(TxnHandler.java:1710)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.unlock(TxnHandler.java:501)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.unlock(HiveMetaStore.java:5571)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy7.unlock(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.unlock(HiveMetaStoreClient.java:1876)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy8.unlock(Unknown Source)
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.unlock(DbLockManager.java:134)
at
[jira] [Updated] (HIVE-10968) Windows: analyze json table via beeline failed throwing Class org.apache.hive.hcatalog.data.JsonSerDe not found
[
https://issues.apache.org/jira/browse/HIVE-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-10968:
--
Fix Version/s: 1.2.1
Windows: analyze json table via beeline failed throwing Class
org.apache.hive.hcatalog.data.JsonSerDe not found
---
Key: HIVE-10968
URL: https://issues.apache.org/jira/browse/HIVE-10968
Project: Hive
Issue Type: Bug
Components: HiveServer2
Environment: Windows
Reporter: Takahiko Saito
Assignee: Hari Sankar Sivarama Subramaniyan
Fix For: 1.2.1
Run the following via beeline:
{noformat}0: jdbc:hive2://localhost:10001 analyze table all100kjson compute
statistics;
15/06/05 20:44:11 INFO log.PerfLogger: PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver
15/06/05 20:44:11 INFO parse.ParseDriver: Parsing command: analyze table
all100kjson compute statistics
15/06/05 20:44:11 INFO parse.ParseDriver: Parse Completed
15/06/05 20:44:11 INFO log.PerfLogger: /PERFLOG method=parse
start=1433537051075 end=1433537051077 duration=2 from=org.
apache.hadoop.hive.ql.Driver
15/06/05 20:44:11 INFO log.PerfLogger: PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Invoking analyze on
original query
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Starting Semantic
Analysis
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Completed phase 1
of Semantic Analysis
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Get metadata for
source tables
15/06/05 20:44:11 INFO metastore.HiveMetaStore: 5: get_table : db=default
tbl=all100kjson
15/06/05 20:44:11 INFO HiveMetaStore.audit: ugi=hadoopqa
ip=unknown-ip-addr cmd=get_table : db=default tbl=a
ll100kjson
15/06/05 20:44:11 INFO metastore.HiveMetaStore: 5: get_table : db=default
tbl=all100kjson
15/06/05 20:44:11 INFO HiveMetaStore.audit: ugi=hadoopqa
ip=unknown-ip-addr cmd=get_table : db=default tbl=a
ll100kjson
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Get metadata for
subqueries
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Get metadata for
destination tables
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Completed getting
MetaData in Semantic Analysis
15/06/05 20:44:11 INFO common.FileUtils: Creating directory if it doesn't
exist: hdfs://dal-hs211:8020/user/hcat/tests/d
ata/all100kjson/.hive-staging_hive_2015-06-05_20-44-11_075_4520028480897676073-5
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Set stats
collection dir : hdfs://dal-hs211:8020/user/hcat/tes
ts/data/all100kjson/.hive-staging_hive_2015-06-05_20-44-11_075_4520028480897676073-5/-ext-1
15/06/05 20:44:11 INFO ppd.OpProcFactory: Processing for TS(5)
15/06/05 20:44:11 INFO log.PerfLogger: PERFLOG method=partition-retrieving
from=org.apache.hadoop.hive.ql.optimizer.ppr
.PartitionPruner
15/06/05 20:44:11 INFO log.PerfLogger: /PERFLOG method=partition-retrieving
start=1433537051345 end=1433537051345 durat
ion=0 from=org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner
15/06/05 20:44:11 INFO metastore.HiveMetaStore: 5: get_indexes : db=default
tbl=all100kjson
15/06/05 20:44:11 INFO HiveMetaStore.audit: ugi=hadoopqa
ip=unknown-ip-addr cmd=get_indexes : db=default tbl
=all100kjson
15/06/05 20:44:11 INFO metastore.HiveMetaStore: 5: get_indexes : db=default
tbl=all100kjson
15/06/05 20:44:11 INFO HiveMetaStore.audit: ugi=hadoopqa
ip=unknown-ip-addr cmd=get_indexes : db=default tbl
=all100kjson
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Looking for table
scans where optimization is applicable
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Found 0 null table
scans
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Looking for table
scans where optimization is applicable
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Found 0 null table
scans
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Looking for table
scans where optimization is applicable
15/06/05 20:44:11 INFO physical.NullScanTaskDispatcher: Found 0 null table
scans
15/06/05 20:44:11 INFO physical.Vectorizer: Validating MapWork...
15/06/05 20:44:11 INFO physical.Vectorizer: Input format:
org.apache.hadoop.mapred.TextInputFormat, doesn't provide vect
orized input
15/06/05 20:44:11 INFO parse.ColumnStatsSemanticAnalyzer: Completed plan
generation
15/06/05 20:44:11 INFO ql.Driver: Semantic Analysis Completed
15/06/05 20:44:11 INFO log.PerfLogger: /PERFLOG method=semanticAnalyze
start=1433537051077 end=1433537051367 duration=2
90 from=org.apache.hadoop.hive.ql.Driver
15/06/05
[jira] [Updated] (HIVE-10857) Accumulo storage handler fail throwing java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token class: class org.apache.accumulo.core.client.security
[
https://issues.apache.org/jira/browse/HIVE-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takahiko Saito updated HIVE-10857:
--
Summary: Accumulo storage handler fail throwing
java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token
class: class org.apache.accumulo.core.client.security.tokens.PasswordToken
(was: Accumulo storage handler fail throwing throwing
java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token
class: class org.apache.accumulo.core.client.security.tokens.PasswordToken)
Accumulo storage handler fail throwing java.lang.IllegalArgumentException:
Cannot determine SASL mechanism for token class: class
org.apache.accumulo.core.client.security.tokens.PasswordToken
---
Key: HIVE-10857
URL: https://issues.apache.org/jira/browse/HIVE-10857
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Takahiko Saito
Assignee: Josh Elser
Fix For: 1.2.1
create table Accumulo storage with Accumulo storage handler fails due to
ACCUMULO-2815.
{noformat}
create table accumulo_1(key string, age int) stored by
'org.apache.hadoop.hive.accumulo.AccumuloStorageHandler' with serdeproperties
( accumulo.columns.mapping = :rowid,info:age);
{noformat}
The error shows:
{noformat}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:org.apache.accumulo.core.client.AccumuloException:
java.lang.IllegalArgumentException: Cannot determine SASL mechanism for token
class: class org.apache.accumulo.core.client.security.tokens.PasswordToken
at
org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:67)
at
org.apache.accumulo.core.client.impl.ConnectorImpl.init(ConnectorImpl.java:67)
at
org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:248)
at
org.apache.hadoop.hive.accumulo.AccumuloConnectionParameters.getConnector(AccumuloConnectionParameters.java:125)
at
org.apache.hadoop.hive.accumulo.AccumuloConnectionParameters.getConnector(AccumuloConnectionParameters.java:111)
at
org.apache.hadoop.hive.accumulo.AccumuloStorageHandler.preCreateTable(AccumuloStorageHandler.java:245)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:664)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:657)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy5.createTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:714)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4135)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
at
org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
