[jira] [Updated] (HIVE-17021) Support replication of concatenate operation.
[ https://issues.apache.org/jira/browse/HIVE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-17021: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Patch pushed to master. > Support replication of concatenate operation. > - > > Key: HIVE-17021 > URL: https://issues.apache.org/jira/browse/HIVE-17021 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17021.01.patch > > > We need to handle cases like ALTER TABLE ... CONCATENATE that also change the > files on disk, and potentially treat them similar to INSERT OVERWRITE, as it > does something equivalent to a compaction. > Note that a ConditionalTask might also be fired at the end of inserts at the > end of a tez task (or other exec engine) if appropriate HiveConf settings are > set, to automatically do this operation - these also need to be taken care of > for replication. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17021) Support replication of concatenate operation.
[ https://issues.apache.org/jira/browse/HIVE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17021: Status: Patch Available (was: Open) > Support replication of concatenate operation. > - > > Key: HIVE-17021 > URL: https://issues.apache.org/jira/browse/HIVE-17021 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17021.01.patch > > > We need to handle cases like ALTER TABLE ... CONCATENATE that also change the > files on disk, and potentially treat them similar to INSERT OVERWRITE, as it > does something equivalent to a compaction. > Note that a ConditionalTask might also be fired at the end of inserts at the > end of a tez task (or other exec engine) if appropriate HiveConf settings are > set, to automatically do this operation - these also need to be taken care of > for replication. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17021) Support replication of concatenate operation.
[ https://issues.apache.org/jira/browse/HIVE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17021: Attachment: HIVE-17021.01.patch Added 01.patch with test cases to verify concatenate operations. - Concatenate operation either from ALTER TABLE or ConditionalTask prepares the plan MergeOperator->MoveTask - MergeOperator, merge all the files from the oven input path and push the output merged file to the temporary staging directory. - MoveTask, moves the merged file from temporary directory to the final warehouse data location. This task uses loadTable and loadPartition methods to load data from temp path to the warehouse which is basically used by Insert Overwrite flow. - Hence, CM recycle and firing insert event done already in the existing code. - Just added test cases to verify it. Request [~anishek]/[~daijy]/[~sushanth]/[~thejas] to review! > Support replication of concatenate operation. > - > > Key: HIVE-17021 > URL: https://issues.apache.org/jira/browse/HIVE-17021 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-17021.01.patch > > > We need to handle cases like ALTER TABLE ... CONCATENATE that also change the > files on disk, and potentially treat them similar to INSERT OVERWRITE, as it > does something equivalent to a compaction. > Note that a ConditionalTask might also be fired at the end of inserts at the > end of a tez task (or other exec engine) if appropriate HiveConf settings are > set, to automatically do this operation - these also need to be taken care of > for replication. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17021) Support replication of concatenate operation.
[ https://issues.apache.org/jira/browse/HIVE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17021: Labels: DR replication (was: ) > Support replication of concatenate operation. > - > > Key: HIVE-17021 > URL: https://issues.apache.org/jira/browse/HIVE-17021 > Project: Hive > Issue Type: Sub-task > Components: Hive, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR, replication > Fix For: 3.0.0 > > > We need to handle cases like ALTER TABLE ... CONCATENATE that also change the > files on disk, and potentially treat them similar to INSERT OVERWRITE, as it > does something equivalent to a compaction. > Note that a ConditionalTask might also be fired at the end of inserts at the > end of a tez task (or other exec engine) if appropriate HiveConf settings are > set, to automatically do this operation - these also need to be taken care of > for replication. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
