phet commented on code in PR #3518:
URL: https://github.com/apache/gobblin/pull/3518#discussion_r892975231
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/dataset/HiveDatasetDescriptor.java:
##########
@@ -39,34 +39,25 @@
@Slf4j
/**
* As of now, {@link HiveDatasetDescriptor} has same implementation as that of
{@link SqlDatasetDescriptor}.
- * Fields {@link HiveDatasetDescriptor#isPartitioned}, {@link
HiveDatasetDescriptor#partitionColumn} and
- * {@link HiveDatasetDescriptor#partitionFormat} are used for methods 'equals'
and 'hashCode'.
+ * Fields {@link HiveDatasetDescriptor#partitionColumn} and {@link
HiveDatasetDescriptor#partitionFormat}
+ * are used for methods 'equals' and 'hashCode'.
*/
@EqualsAndHashCode (exclude = {"whitelistBlacklist"}, callSuper = true)
public class HiveDatasetDescriptor extends SqlDatasetDescriptor {
- static final String IS_PARTITIONED_KEY = "isPartitioned";
static final String PARTITION_COLUMN = "partition.column";
static final String PARTITION_FORMAT = "partition.format";
static final String CONFLICT_POLICY = "conflict.policy";
- private final boolean isPartitioned;
private final String partitionColumn;
private final String partitionFormat;
private final String conflictPolicy;
WhitelistBlacklist whitelistBlacklist;
public HiveDatasetDescriptor(Config config) throws IOException {
super(config);
- this.isPartitioned = ConfigUtils.getBoolean(config, IS_PARTITIONED_KEY,
true);
- if (isPartitioned) {
- partitionColumn = ConfigUtils.getString(config, PARTITION_COLUMN,
DatePartitionHiveVersionFinder.DEFAULT_PARTITION_KEY_NAME);
- partitionFormat = ConfigUtils.getString(config, PARTITION_FORMAT,
DatePartitionHiveVersionFinder.DEFAULT_PARTITION_VALUE_DATE_TIME_PATTERN);
- conflictPolicy =
HiveCopyEntityHelper.ExistingEntityPolicy.REPLACE_PARTITIONS.name();
- } else {
- partitionColumn = "";
- partitionFormat = "";
- conflictPolicy =
HiveCopyEntityHelper.ExistingEntityPolicy.REPLACE_TABLE.name();
- }
+ partitionColumn = ConfigUtils.getString(config, PARTITION_COLUMN,
DatePartitionHiveVersionFinder.DEFAULT_PARTITION_KEY_NAME);
+ partitionFormat = ConfigUtils.getString(config, PARTITION_FORMAT,
DatePartitionHiveVersionFinder.DEFAULT_PARTITION_VALUE_DATE_TIME_PATTERN);
+ conflictPolicy =
HiveCopyEntityHelper.ExistingEntityPolicy.REPLACE_TABLE_AND_PARTITIONS.name();
Review Comment:
the values I see for these defaults are `"datepartition"`,
`"yyyy-MM-dd-HH"`, so not backwards compatible w/ the `""` and `""` previously
when `!isPartitioned`.
could that cause an issue? e.g. when the user doesn't use partitioning and
therefore gave nothing, so we fell back to `"datepartition"`, how would we
discern from when they pass that value explicitly with the intent to use
partitioning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
