rbalamohan commented on a change in pull request #1060:
URL: https://github.com/apache/hive/pull/1060#discussion_r435785601
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
##########
@@ -154,7 +157,45 @@ private ReplicationState initialReplicationState() throws
SemanticException {
);
}
+ private boolean isImmutableDataCopy() {
+ //at the time of repl dump, data got referenced externally and not part of
the dump.
+ return HiveConf.getBoolVar(context.hiveConf,
REPL_DUMP_SKIP_IMMUTABLE_DATA_COPY);
+ }
+
+ /**
+ * Get all partitions and consolidate them into single partition request.
+ * Also, copy relevant stats and other information from original request.
+ *
+ * @throws SemanticException
+ */
+ private void addConsolidatedPartitionDesc() throws Exception {
+ List<AlterTableAddPartitionDesc.PartitionDesc> partitions = new
LinkedList<>();
+ for (AlterTableAddPartitionDesc alterTableAddPartitionDesc :
event.partitionDescriptions(tableDesc)) {
+
+ AlterTableAddPartitionDesc.PartitionDesc src =
alterTableAddPartitionDesc.getPartitions().get(0);
+
+ partitions.add(new AlterTableAddPartitionDesc.PartitionDesc(
+ src.getPartSpec(), src.getLocation(), src.getPartParams(),
src.getInputFormat(),
+ src.getOutputFormat(), src.getNumBuckets(), src.getCols(),
src.getSerializationLib(),
+ src.getSerdeParams(), src.getBucketCols(), src.getSortCols(),
src.getColStats(),
+ src.getWriteId()));
+ }
+ AlterTableAddPartitionDesc consolidatedPartitionDesc = new
AlterTableAddPartitionDesc(tableDesc.getDatabaseName(),
+ tableDesc.getTableName(), true, partitions);
+
+ addPartition(false, consolidatedPartitionDesc, null);
+ if (partitions.size() > 0) {
+ LOG.info("Added {} partitions", partitions.size());
+ }
+ }
+
private TaskTracker forNewTable() throws Exception {
+ if (isImmutableDataCopy()) {
Review comment:
Since we rely on "REPL_DUMP_SKIP_IMMUTABLE_DATA_COPY" in both dump/load,
I haven't added that config here. We didn't honor metadata only dump anyways
earlier https://issues.apache.org/jira/browse/HIVE-23499
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
