[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009287#comment-16009287
 ] 

ASF GitHub Bot commented on HIVE-13652:
---

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/180


> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, TODOC3.0, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1600#comment-1600
 ] 

Sankar Hariappan commented on HIVE-13652:
-

Thanks [~thejas] for the commit!

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, TODOC3.0, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002170#comment-16002170
 ] 

Lefty Leverenz commented on HIVE-13652:
---

Okay, I added a TODOC3.0 label.  Thanks Thejas.

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, TODOC3.0, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002164#comment-16002164
 ] 

Thejas M Nair commented on HIVE-13652:
--

Yes, I guess we could document this as a known issue in earlier versions.


> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002154#comment-16002154
 ] 

Lefty Leverenz commented on HIVE-13652:
---

Does this need to be documented in the wiki?

* [Import/Export | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002138#comment-16002138
 ] 

Thejas M Nair commented on HIVE-13652:
--

Thanks for the updates to the test case. +1


> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000459#comment-16000459
 ] 

Sankar Hariappan commented on HIVE-13652:
-

[~thejas]
The test failures are irrelevant to this patch. Please help to commit this 
patch.

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999480#comment-15999480
 ] 

Hive QA commented on HIVE-13652:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866633/HIVE-13652.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10653 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30]
 (batchId=148)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5078/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5078/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5078/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866633 - PreCommit-HIVE-Build

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998876#comment-15998876
 ] 

Hive QA commented on HIVE-13652:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12866633/HIVE-13652.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10653 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5051/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5051/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5051/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12866633 - PreCommit-HIVE-Build

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998870#comment-15998870
 ] 

Thejas M Nair commented on HIVE-13652:
--

Created HIVE-16597 to track larger cleanup around datastructure for partSpec.


> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998857#comment-15998857
 ] 

Thejas M Nair commented on HIVE-13652:
--

[~luky]
LinkedHashMap has iteration order that is same as the insertion order, so this 
would work. So the ordering is defined (it uses a doubly-linked list to 
remember the order).

However, I agree that the use of Map for partSpec object in the 
AddPartitionDesc methods makes it vulnerable to such bugs again in future. We 
should do a cleanup to replace use of Map with something like 
List> which would have prevented this bug in first place. 
I will create a follow up bug for that.



> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Lukas Waldmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998489#comment-15998489
 ] 

Lukas Waldmann commented on HIVE-13652:
---

I just wonder, if by using LinkedHashMap we don't introduce just another 
vulnerability there as order of items in the HashMap is not defined and if ever 
hashing algo change than again we may get  completely different order?




> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998464#comment-15998464
 ] 

Sankar Hariappan commented on HIVE-13652:
-

Updated patch for additional verification of partition location.
[~thejas], Please review the patch.

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997792#comment-15997792
 ] 

Sankar Hariappan commented on HIVE-13652:
-

[~thejas] 
Yes, I knew the test will pass even without this change. I just added the test 
as this case was never there.
But, I was not aware (or at least didn't click) of above show commands to 
verify it. Will use these commands to verify the fix and provide a patch soon.
Thanks!

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997495#comment-15997495
 ] 

Thejas M Nair commented on HIVE-13652:
--

[~sankarh]
It seems like this test would pass even without this change. Can you also add a 
check to verify the partition name and location after repl-load ?

ie, check output of "show partitions" and
the location filed (suffix) in output of  "show table extended like" + dbName + 
"_dupe.namelist partition (year=1990 and month=5 and day=25)" ?



> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997249#comment-15997249
 ] 

ASF GitHub Bot commented on HIVE-13652:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/180

HIVE-13652: Import table change order of dynamic partitions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-13652

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/180.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #180


commit 3af1f73aeb854c5b911f48c925365336f15cb0e5
Author: Sankar Hariappan 
Date:   2017-05-04T18:51:38Z

HIVE-13652: Import table change order of dynamic partitions




> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>  Components: Import/Export, repl
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>Assignee: Sankar Hariappan
>  Labels: DR, replication
> Attachments: HIVE-13652.01.patch, 
> ReplLoad_PartitionOrder_AfterFix_Log.png, 
> ReplLoad_PartitionOrder_BeforeFix_Log.png
>
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285104#comment-15285104
 ] 

Sushanth Sowmyan commented on HIVE-13652:
-

> In any case current behaviour is the worst possible as it leaves partitioning 
> on "higher will"

I laughed at this, and you're definitely right about this.

The rest is merely context behind why this problem has not been tackled earlier 
(since we break it for some users at the very least), and honestly, if we don't 
tackle it now, we simply kick the ball further down the road, and it will need 
tackling one way or another, which we have been doing so far.

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13652) Import table change order of dynamic partitions

[Lukas Waldmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272635#comment-15272635
 ] 

Lukas Waldmann commented on HIVE-13652:
---

I just wonder how come there is no definition of order of partitions - when you 
create table you define order in "partition by" clause, when you do "describe 
table" than you get the partitioning columns in right order (even after "wrong" 
import).
If you are worried about external tables than lets use your b) option from 
other solutions, even though I don't fully comprehend how would it not help 
existing export/import cases

In any case current behaviour is the worst possible as it leaves partitioning 
on "higher will"

> Import table change order of dynamic partitions
> ---
>
> Key: HIVE-13652
> URL: https://issues.apache.org/jira/browse/HIVE-13652
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Lukas Waldmann
>
> Table with multiple dynamic partitions like year,month, day exported using 
> "export table" command is imported (using "import table") such a way that 
> order of partitions is changed to day, month, year.
> Export DB:  Hive 0.14
> Import DB:  Hive 1.2.1000.2.4.0.0-169
> Tables created as:
> create table T1
> ( ... ) PARTITIONED BY (period_year string, period_month string, period_day 
> string) STORED AS ORC TBLPROPERTIES ("orc.compress"="SNAPPY");
> export command:
> export table t1 to 'path'
> import command:
> import table t1 from 'path'
> HDFS file structure on both original table location and export path keeps the 
> original partition order ../year/month/day
> HDFS file structure after import is .../day/month/year



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)