[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153172#comment-16153172 ] Lefty Leverenz commented on HIVE-12767: --- The documentation was reverted (thanks, [~zsombor.klara]) so I'll remove the TODOC2.4 label. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077227#comment-16077227 ] Lefty Leverenz commented on HIVE-12767: --- Added a TODOC2.4 label in case we need to revert the documentation (3 wikidocs listed in comment 19/May/17). But let's wait a while and see how the issue is solved. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.4 > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016900#comment-16016900 ] Lefty Leverenz commented on HIVE-12767: --- Thanks for the docs, [~zsombor.klara]. I changed the version numbers from 2.2.0 to 2.4.0 in the wiki and removed the TODOC2.2 label from this issue. Here are the doc links again: * [Parquet -- Hive2.4.0 | https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Hive2.4.0] * [Configuration Properties -- hive.parquet.mr.int96.enable.utc.write.zone | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.parquet.mr.int96.enable.utc.write.zone] * [DDL -- TBLPROPERTIES | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-listTableProperties] > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Fix For: 2.4.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891949#comment-15891949 ] Barna Zsombor Klara commented on HIVE-12767: I've raised HIVE-16088 to fix the misnamed configuration property. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889906#comment-15889906 ] Barna Zsombor Klara commented on HIVE-12767: Thanks [~leftylev] for the help, I updated the docs you mentioned. As for the "hive." prefix, I don't know if this is a strict convention or not. While most configuration properties start with the "hive" prefix, others do not, like "parquet.memory.pool.ratio", and I can't see a clear rule clarifying when the prefix is needed. [~spena] should we (should I) rename it or was is meant to be like this? > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889747#comment-15889747 ] Lefty Leverenz commented on HIVE-12767: --- Here's where the documentation belongs: * [Parquet | https://cwiki.apache.org/confluence/display/Hive/Parquet] * [Configuration Properties -- Parquet | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Parquet] for *parquet.mr.int96.enable.utc.write.zone* * [Table Properties | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-listTableProperties] Question: Shouldn't the config name start with "hive." as in "hive.parquet.mr.int96.enable.utc.write.zone"? > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886739#comment-15886739 ] Sergio Peña commented on HIVE-12767: +1 Thanks [~zsombor.klara]. I think the patch is good now. Thanks for continuing the work on this patch. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch, > HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch, > HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch, > HIVE-12767.9.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883733#comment-15883733
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854485/HIVE-12767.11.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10274 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854485 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.10.patch, HIVE-12767.11.patch,
> HIVE-12767.3.patch, HIVE-12767.4.patch, HIVE-12767.5.patch,
> HIVE-12767.6.patch, HIVE-12767.7.patch, HIVE-12767.8.patch,
> HIVE-12767.9.patch, TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878798#comment-15878798 ] Sergio Peña commented on HIVE-12767: Great, thanks [~zsombor.klara] for continuing on this work. I just left a couple of comments on the RB. > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch, > HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch, > HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch, > TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863951#comment-15863951
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852366/HIVE-12767.10.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10236 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=101)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3520/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3520/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3520/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852366 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch,
> HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch,
> HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch,
> TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863788#comment-15863788
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852362/HIVE-12767.10.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10250 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3518/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3518/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3518/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852362 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch,
> HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch,
> HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch,
> TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863723#comment-15863723
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852347/HIVE-12767.9.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10250 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3517/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3517/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3517/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852347 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.10.patch, HIVE-12767.3.patch,
> HIVE-12767.4.patch, HIVE-12767.5.patch, HIVE-12767.6.patch,
> HIVE-12767.7.patch, HIVE-12767.8.patch, HIVE-12767.9.patch,
> TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861366#comment-15861366 ] Barna Zsombor Klara commented on HIVE-12767: query14 and encryption_join_with_different_encryption_keys are know flaky tests (HIVE-15744, HIVE-15696) > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch, > HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch, > HIVE-12767.8.patch, TestNanoTimeUtils.java > > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861350#comment-15861350
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852050/HIVE-12767.8.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10258 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3490/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3490/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3490/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852050 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch,
> HIVE-12767.8.patch, TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860144#comment-15860144
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851905/HIVE-12767.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3471/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3471/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3471/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-09 20:31:57.050
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3471/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-09 20:31:57.053
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 36ff484 HIVE-15851 CompactorMR.launchCompactionJob() should use
JobClient.submitJob() not runJob (Eugene Koifman, reviewed by Wei Zheng)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 36ff484 HIVE-15851 CompactorMR.launchCompactionJob() should use
JobClient.submitJob() not runJob (Eugene Koifman, reviewed by Wei Zheng)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-09 20:31:58.075
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java:127
error:
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851905 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch,
> HIVE-12767.8.patch, TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858175#comment-15858175
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851654/HIVE-12767.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3442/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-08 16:13:13.361
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3442/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-08 16:13:13.364
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 8a06b9e HIVE-15626: beeline should not exit after canceling the
query on ctrl-c (Vihang Karajgaonkar via Chaoyu Tang)
+ git clean -f -d
Removing
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RepartitionShuffler.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 8a06b9e HIVE-15626: beeline should not exit after canceling the
query on ctrl-c (Vihang Karajgaonkar via Chaoyu Tang)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-08 16:13:14.446
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git diff header lacks filename information when removing 0 leading
pathname components (line 20)
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851654 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> HIVE-12767.5.patch, HIVE-12767.6.patch, HIVE-12767.7.patch,
> TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856234#comment-15856234
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851409/HIVE-12767.6.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10142 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=17)
[list_bucket_dml_1.q,ppd_join3.q,auto_join23.q,list_bucket_dml_11.q,join10.q,avro_type_evolution.q,create_struct_table.q,cbo_const.q,skewjoin_mapjoin9.q,hook_context_cs.q,subquery_unqualcolumnrefs.q,exim_22_import_exist_authsuccess.q,groupby1.q,cbo_rp_udf_udaf.q,udf_regexp_replace.q,vector_decimal_aggregate.q,authorization_grant_public_role.q,partition_wise_fileformat.q,sort_merge_join_desc_5.q,union_ppr.q,spark_combine_equivalent_work.q,stats_partial_size.q,partition_date2.q,join32.q,list_bucket_dml_14.q,input34.q,insert_values_acid_not_bucketed.q,udf_parse_url.q,schema_evol_text_nonvec_part.q,ctas_char.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=18)
[encryption_move_tbl.q,cbo_rp_udf_percentile2.q,udf_lower.q,union_remove_11.q,transform_acid.q,constprog_dp.q,join45.q,subquery_multi.q,orc_merge1.q,groupby_multi_single_reducer2.q,offset_limit_global_optimizer.q,dbtxnmgr_query4.q,multi_column_in.q,create_func1.q,create_skewed_table1.q,authorization_view_1.q,show_tables.q,orc_empty_files.q,ba_table2.q,acid_vectorization_project.q,union_remove_19.q,vector_decimal_3.q,stats3.q,update_where_no_match.q,parquet_join.q,exim_24_import_nonexist_authsuccess.q,outer_join_ppr.q,quotedid_partition.q,ansi_sql_arithmetic.q,join26.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=19)
[cp_mj_rc.q,order.q,udf_bitwise_shiftleft.q,groupby1_limit.q,insert_values_non_partitioned.q,skewjoinopt10.q,extrapolate_part_stats_date.q,udf_sin.q,vectorized_math_funcs.q,vectorization_2.q,join14.q,query_result_fileformat.q,udf3.q,cbo_union_view.q,drop_function.q,tez_union_decimal.q,exim_23_import_part_authsuccess.q,load_dyn_part10.q,nonmr_fetch.q,order_null.q,cbo_rp_views.q,lvj_mapjoin.q,insert_acid_dynamic_partition.q,skewjoinopt6.q,cbo_rp_cross_product_check_2.q,mapreduce7.q,update_two_cols.q,correlationoptimizer11.q,stats_empty_partition.q,authorization_2.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=22)
[multi_insert_mixed.q,specialChar.q,join_cond_pushdown_3.q,cast_on_constant.q,union_remove_10.q,skewjoinopt21.q,input_part0.q,vector_join_nulls.q,combine2_hadoop20.q,infer_join_preds.q,udf_hex.q,reducesink_dedup.q,insert1.q,input16.q,udf_in_file.q,vector_empty_where.q,udf_variance.q,join42.q,auto_join12.q,groupby_sort_2.q,subquery_alias.q,cte_mat_3.q,parenthesis_star_by.q,vector_decimal_round_2.q,udf_conv.q,column_names_with_leading_and_trailing_spaces.q,vectorized_mapjoin2.q,union_stats.q,nullgroup4.q,authorization_view_disable_cbo_2.q]
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_int96_timestamp]
(batchId=33)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3417/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3417/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3417/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851409 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> HIVE-12767.5.patch, HIVE-12767.6.patch, TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854565#comment-15854565
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851198/HIVE-12767.5.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10224 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=100)
[skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,join_reorder.q,sample1.q,bucketmapjoin8.q,union34.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_external_time]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_int96_timestamp]
(batchId=33)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testWaitQueuePreemption
(batchId=278)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3396/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3396/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3396/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12851198 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> HIVE-12767.5.patch, TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850285#comment-15850285
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12786152/TestNanoTimeUtils.java
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3331/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3331/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3331/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-02 18:33:04.296
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3331/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-02 18:33:04.300
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 7cca097 HIVE-14420: Fix orc_llap_counters.q test failure in
master (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/parquet_ppd_multifiles.q
Removing ql/src/test/results/clientpositive/parquet_ppd_multifiles.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 7cca097 HIVE-14420: Fix orc_llap_counters.q test failure in
master (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-02 18:33:05.296
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12786152 - PreCommit-HIVE-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch,
> TestNanoTimeUtils.java
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
>
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131453#comment-15131453
]
Ryan Blue commented on HIVE-12767:
--
[~spena], the q tests from the last patch look great and make me confident that
the create table and property behavior are correct. I still think that the time
zone conversion isn't implementing what it should (see below). I also found a
couple of minor things:
# Minor: The CTAS test says the zone property on the resulting table should be
PST, but is is (and should be) UTC. I think the comment is wrong and the
behavior is correct.
# It would be nice to validate that the table property always overrides the
global option
# Is there a test that files are created with the table property in key/value
metadata? I think that the copy from one table to another with a different
property set wouldn't work otherwise, but I want to make sure.
# What is the Hive 2.0 test doing? It looks like it is validating that setting
the table property to UTC converts values back to local time? Or was the file
written with UTC? (If so, we should have one written in another zone.)
The Hive 2.0 test I think demonstrates that the time zone conversion isn't
correct. It is correct for the with and without conversion cases. But, it isn't
a correct implementation for the meaning of the table property's time zone. In
the Hive 2.0 test, it is set to UTC, but the value is read in local (if I'm
reading this correctly).
Here's the problem: the back-end converts julian/nanos to year/month/day/hour
values, then sets those on a calendar. It then uses the backing time in
milliseconds to create a Timestamp. That timestamp is in _local_ time because
of HIVE-12192, so to have the time _not_ adjusted, the calendar needs to also
use the local zone. So using the local zone ends up not adjusting the value.
The problem is that the current patch uses this implementation to adjust values
to the table property zone. That doesn't work because the "no offset" zone for
the table property is UTC, not the local zone. I'm attaching a file that
demonstrates the problem. It uses [Spark's implementation of
fromJulianDay|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L180]
to demonstrate what I'm talking about. I chose a date/time at random and
tested out the two implementations.
Tests that show _correct_ behavior:
* {{testKnownTimestampWithoutConversion}} shows that when UTC is used, the
fromJulianDay value is the same as the "skip conversion" value returned by
NanoTimeUtils.
* {{testKnownTimestampWithConversion}} shows that when the local zone is used
(for me, PST), the fromJulianDay value is the same as the "do not skip
conversion" value returned by NanoTimeUtils.
Tests that show _incorrect_ behavior:
* {{testKnownWithZoneArgumentUTC}} shows that passing a UTC calendar to
NanoTimeUtils does not match the fromJulianDay/UTC value that matched the "skip
conversion" in tests above. This doesn't pass, but should.
* {{testKnownWithWrongZone}} shows that passing a PST calendar to NanoTimeUtils
matches the fromJulianDay/UTC value that corresponds to "skip conversion". This
shouldn't pass, but does.
I'm attaching my test to this and can help if you have any questions about it.
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097385#comment-15097385
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12782125/HIVE-12767.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10005 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6616/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6616/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6616/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12782125 - PreCommit-HIVE-TRUNK-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-12767.3.patch, HIVE-12767.4.patch
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent. This keeps the data in the table consistent.
> * New tables created by Hive will set the table property to UTC if the global
> option to set the property for new tables is enabled.
> ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set
> the property unless the global setting to do so is enabled.
> ** Tables created using CREATE TABLE LIKE will copy the
> property of the table that is copied.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[
https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086549#comment-15086549
]
Hive QA commented on HIVE-12767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12780780/HIVE-12767.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6533/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6533/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6533/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6533/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
05e6096..95f2bd8 branch-1 -> origin/branch-1
8069b59..cb17456 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 8069b59 HIVE-12597 : LLAP - allow using elevator without cache
(Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/parquet_join2.q
Removing ql/src/test/results/clientpositive/parquet_join2.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at cb17456 HIVE-12796: Switch to 32-bits containers for HMS upgrade
tests.
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12780780 - PreCommit-HIVE-TRUNK-Build
> Implement table property to address Parquet int96 timestamp bug
> ---
>
> Key: HIVE-12767
> URL: https://issues.apache.org/jira/browse/HIVE-12767
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-12767.2.patch
>
>
> Parque timestamps using INT96 are not compatible with other tools, like
> Impala, due to issues in Hive because it adjusts timezones values in a
> different way than Impala.
> To address such issues. a new table property (parquet.mr.int96.write.zone)
> must be used in Hive that detects what timezone to use when writing and
> reading timestamps from Parquet.
> The following is the exit criteria for the fix:
> * Hive will read Parquet MR int96 timestamp data and adjust values using a
> time zone from a table property, if set, or using the local time zone if it
> is absent. No adjustment will be applied to data written by Impala.
> * Hive will write Parquet int96 timestamps using a time zone adjustment from
> the same table property, if set, or using the local time zone if it is
> absent.
[jira] [Commented] (HIVE-12767) Implement table property to address Parquet int96 timestamp bug
[ https://issues.apache.org/jira/browse/HIVE-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075538#comment-15075538 ] Sergio Peña commented on HIVE-12767: cc: [~kamrul] > Implement table property to address Parquet int96 timestamp bug > --- > > Key: HIVE-12767 > URL: https://issues.apache.org/jira/browse/HIVE-12767 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña > > Parque timestamps using INT96 are not compatible with other tools, like > Impala, due to issues in Hive because it adjusts timezones values in a > different way than Impala. > To address such issues. a new table property (parquet.mr.int96.write.zone) > must be used in Hive that detects what timezone to use when writing and > reading timestamps from Parquet. > The following is the exit criteria for the fix: > * Hive will read Parquet MR int96 timestamp data and adjust values using a > time zone from a table property, if set, or using the local time zone if it > is absent. No adjustment will be applied to data written by Impala. > * Hive will write Parquet int96 timestamps using a time zone adjustment from > the same table property, if set, or using the local time zone if it is > absent. This keeps the data in the table consistent. > * New tables created by Hive will set the table property to UTC if the global > option to set the property for new tables is enabled. > ** Tables created using CREATE TABLE and CREATE TABLE LIKE FILE will not set > the property unless the global setting to do so is enabled. > ** Tables created using CREATE TABLE LIKE will copy the > property of the table that is copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
