subject:"\[jira\] \[Updated\] \(DRILL\-6969\) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled"

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

2019-01-22 Thread Volodymyr Vysotskyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi updated DRILL-6969:
---
Description: 
Steps to reproduce:
0. Set PST timezone.
1. Create the table in MaprDB shell:
{code:java}
create /tmp/testtimestamp
insert /tmp/testtimestamp --value '{"_id":"1","datestring":"2018-01-01 
12:12:12.123","datetimestamp":{"$date":"2018-01-01T20:12:12.123Z"}}'
insert /tmp/testtimestamp --value '{"_id":"2","datestring":"-12-31 
23:59:59.999","datetimestamp":{"$date":"1-01-01T07:59:59.999Z"}}'
{code}
2. Create a hive table:
{code:sql}
create external table `testtimestamp` (`_id` string, datestring string, 
datetimestamp timestamp)
ROW FORMAT SERDE 'org.apache.hadoop.hive.maprdb.json.serde.MapRDBSerDe'
STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
TBLPROPERTIES ( 'maprdb.column.id'='_id', 
'maprdb.table.name'='/tmp/testtimestamp');
{code}
3. Disable native reader and run the query on the table from Drill using hive 
plugin:
{code:sql}
alter session set store.hive.maprdb_json.optimize_scan_with_native_reader=false;
select * from hive.testtimestamp;
{code}
It returns:
{noformat}
+--+--+--+
| _id  |datestring|  datetimestamp   |
+--+--+--+
| 1| 2018-01-01 12:12:12.123  | 2018-01-01 12:12:12.123  |
| 2| -12-31 23:59:59.999  | -12-31 23:59:59.999  |
+--+--+--+
{noformat}
4. Enable native reader and run the query on the same table:
{code:sql}
alter session set store.hive.maprdb_json.optimize_scan_with_native_reader=true;
select * from hive.testtimestamp;
{code}
It returns:
{noformat}
+--+--+---+
| _id  |datestring|   datetimestamp   |
+--+--+---+
| 1| 2018-01-01 12:12:12.123  | 2018-01-01 20:12:12.123   |
| 2| -12-31 23:59:59.999  | 1-01-01 07:59:59.999  |
+--+--+---+
{noformat}
h2. For documentation:

Added the following Mapr-DB Format Setting:
||Option||Description||Value||
|readTimestampWithZoneOffset|When enabled, Drill converts timestamp values read 
form MapR Database from UTC to local timezone. Disabled by default.|true\|false|

Added the following configuration option:
||Name||Default||Description||
|store.hive.maprdb_json.read_timestamp_with_timezone_offset|FALSE|Enables Drill 
to read timestamp values with timezone offset when hive plugin is used and 
Drill native MaprDB JSON reader usage is enabled. (Drill 1.16+)|

  was:
Steps to reproduce:
0. Set PST timezone.
1. Create the table in MaprDB shell:
{code}
create /tmp/testtimestamp
insert /tmp/testtimestamp --value '{"_id":"1","datestring":"2018-01-01 
12:12:12.123","datetimestamp":{"$date":"2018-01-01T20:12:12.123Z"}}'
insert /tmp/testtimestamp --value '{"_id":"2","datestring":"-12-31 
23:59:59.999","datetimestamp":{"$date":"1-01-01T07:59:59.999Z"}}'
{code}
2. Create a hive table:
{code:sql}
create external table `testtimestamp` (`_id` string, datestring string, 
datetimestamp timestamp)
ROW FORMAT SERDE 'org.apache.hadoop.hive.maprdb.json.serde.MapRDBSerDe'
STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
TBLPROPERTIES ( 'maprdb.column.id'='_id', 
'maprdb.table.name'='/tmp/testtimestamp');
{code}
3. Disable native reader and run the query on the table from Drill using hive 
plugin:
{code:sql}
alter session set store.hive.maprdb_json.optimize_scan_with_native_reader=false;
select * from hive.testtimestamp;
{code}
It returns:
{noformat}
+--+--+--+
| _id  |datestring|  datetimestamp   |
+--+--+--+
| 1| 2018-01-01 12:12:12.123  | 2018-01-01 12:12:12.123  |
| 2| -12-31 23:59:59.999  | -12-31 23:59:59.999  |
+--+--+--+
{noformat}
4. Enable native reader and run the query on the same table:
{code:sql}
alter session set store.hive.maprdb_json.optimize_scan_with_native_reader=true;
select * from hive.testtimestamp;
{code}
It returns:
{noformat}
+--+--+---+
| _id  |datestring|   datetimestamp   |
+--+--+---+
| 1| 2018-01-01 12:12:12.123  | 2018-01-01 20:12:12.123   |
| 2| -12-31 23:59:59.999  | 1-01-01 07:59:59.999  |
+--+--+---+
{noformat}


> Inconsistent results when reading MaprDB JSON tables using hive plugin when 
> native reader is enabled
>

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

2019-01-22 Thread Volodymyr Vysotskyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi updated DRILL-6969:
---
Labels: doc-impacting ready-to-commit  (was: ready-to-commit)

> Inconsistent results when reading MaprDB JSON tables using hive plugin when 
> native reader is enabled
> 
>
> Key: DRILL-6969
> URL: https://issues.apache.org/jira/browse/DRILL-6969
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.16.0
>
>
> Steps to reproduce:
> 0. Set PST timezone.
> 1. Create the table in MaprDB shell:
> {code:java}
> create /tmp/testtimestamp
> insert /tmp/testtimestamp --value '{"_id":"1","datestring":"2018-01-01 
> 12:12:12.123","datetimestamp":{"$date":"2018-01-01T20:12:12.123Z"}}'
> insert /tmp/testtimestamp --value '{"_id":"2","datestring":"-12-31 
> 23:59:59.999","datetimestamp":{"$date":"1-01-01T07:59:59.999Z"}}'
> {code}
> 2. Create a hive table:
> {code:sql}
> create external table `testtimestamp` (`_id` string, datestring string, 
> datetimestamp timestamp)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.maprdb.json.serde.MapRDBSerDe'
> STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
> TBLPROPERTIES ( 'maprdb.column.id'='_id', 
> 'maprdb.table.name'='/tmp/testtimestamp');
> {code}
> 3. Disable native reader and run the query on the table from Drill using hive 
> plugin:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=false;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+--+
> | _id  |datestring|  datetimestamp   |
> +--+--+--+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 12:12:12.123  |
> | 2| -12-31 23:59:59.999  | -12-31 23:59:59.999  |
> +--+--+--+
> {noformat}
> 4. Enable native reader and run the query on the same table:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=true;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+---+
> | _id  |datestring|   datetimestamp   |
> +--+--+---+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 20:12:12.123   |
> | 2| -12-31 23:59:59.999  | 1-01-01 07:59:59.999  |
> +--+--+---+
> {noformat}
> h2. For documentation:
> Added the following Mapr-DB Format Setting:
> ||Option||Description||Value||
> |readTimestampWithZoneOffset|When enabled, Drill converts timestamp values 
> read form MapR Database from UTC to local timezone. Disabled by 
> default.|true\|false|
> Added the following configuration option:
> ||Name||Default||Description||
> |store.hive.maprdb_json.read_timestamp_with_timezone_offset|FALSE|Enables 
> Drill to read timestamp values with timezone offset when hive plugin is used 
> and Drill native MaprDB JSON reader usage is enabled. (Drill 1.16+)|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

2019-01-14 Thread Volodymyr Vysotskyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi updated DRILL-6969:
---
Labels: ready-to-commit  (was: )

> Inconsistent results when reading MaprDB JSON tables using hive plugin when 
> native reader is enabled
> 
>
> Key: DRILL-6969
> URL: https://issues.apache.org/jira/browse/DRILL-6969
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> Steps to reproduce:
> 0. Set PST timezone.
> 1. Create the table in MaprDB shell:
> {code}
> create /tmp/testtimestamp
> insert /tmp/testtimestamp --value '{"_id":"1","datestring":"2018-01-01 
> 12:12:12.123","datetimestamp":{"$date":"2018-01-01T20:12:12.123Z"}}'
> insert /tmp/testtimestamp --value '{"_id":"2","datestring":"-12-31 
> 23:59:59.999","datetimestamp":{"$date":"1-01-01T07:59:59.999Z"}}'
> {code}
> 2. Create a hive table:
> {code:sql}
> create external table `testtimestamp` (`_id` string, datestring string, 
> datetimestamp timestamp)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.maprdb.json.serde.MapRDBSerDe'
> STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
> TBLPROPERTIES ( 'maprdb.column.id'='_id', 
> 'maprdb.table.name'='/tmp/testtimestamp');
> {code}
> 3. Disable native reader and run the query on the table from Drill using hive 
> plugin:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=false;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+--+
> | _id  |datestring|  datetimestamp   |
> +--+--+--+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 12:12:12.123  |
> | 2| -12-31 23:59:59.999  | -12-31 23:59:59.999  |
> +--+--+--+
> {noformat}
> 4. Enable native reader and run the query on the same table:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=true;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+---+
> | _id  |datestring|   datetimestamp   |
> +--+--+---+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 20:12:12.123   |
> | 2| -12-31 23:59:59.999  | 1-01-01 07:59:59.999  |
> +--+--+---+
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

2019-01-11 Thread Volodymyr Vysotskyi (JIRA)



 [ 
https://issues.apache.org/jira/browse/DRILL-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi updated DRILL-6969:
---
Reviewer: Aman Sinha

> Inconsistent results when reading MaprDB JSON tables using hive plugin when 
> native reader is enabled
> 
>
> Key: DRILL-6969
> URL: https://issues.apache.org/jira/browse/DRILL-6969
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.16.0
>
>
> Steps to reproduce:
> 0. Set PST timezone.
> 1. Create the table in MaprDB shell:
> {code}
> create /tmp/testtimestamp
> insert /tmp/testtimestamp --value '{"_id":"1","datestring":"2018-01-01 
> 12:12:12.123","datetimestamp":{"$date":"2018-01-01T20:12:12.123Z"}}'
> insert /tmp/testtimestamp --value '{"_id":"2","datestring":"-12-31 
> 23:59:59.999","datetimestamp":{"$date":"1-01-01T07:59:59.999Z"}}'
> {code}
> 2. Create a hive table:
> {code:sql}
> create external table `testtimestamp` (`_id` string, datestring string, 
> datetimestamp timestamp)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.maprdb.json.serde.MapRDBSerDe'
> STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
> TBLPROPERTIES ( 'maprdb.column.id'='_id', 
> 'maprdb.table.name'='/tmp/testtimestamp');
> {code}
> 3. Disable native reader and run the query on the table from Drill using hive 
> plugin:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=false;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+--+
> | _id  |datestring|  datetimestamp   |
> +--+--+--+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 12:12:12.123  |
> | 2| -12-31 23:59:59.999  | -12-31 23:59:59.999  |
> +--+--+--+
> {noformat}
> 4. Enable native reader and run the query on the same table:
> {code:sql}
> alter session set 
> store.hive.maprdb_json.optimize_scan_with_native_reader=true;
> select * from hive.testtimestamp;
> {code}
> It returns:
> {noformat}
> +--+--+---+
> | _id  |datestring|   datetimestamp   |
> +--+--+---+
> | 1| 2018-01-01 12:12:12.123  | 2018-01-01 20:12:12.123   |
> | 2| -12-31 23:59:59.999  | 1-01-01 07:59:59.999  |
> +--+--+---+
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

[jira] [Updated] (DRILL-6969) Inconsistent results when reading MaprDB JSON tables using hive plugin when native reader is enabled

4 matches

Site Navigation

Mail list logo

Footer information