[
https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=394601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394601
]
ASF GitHub Bot logged work on GOBBLIN-1054:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 28/Feb/20 02:08
Start Date: 28/Feb/20 02:08
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2894:
GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r385474038
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##########
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
* </p>
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
- Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
- Preconditions.checkArgument(properties.containsKey(LOOKBACK),
ERROR_MESSAGE);
- Preconditions.checkArgument(properties.containsKey(DATETIME_FORMAT),
ERROR_MESSAGE);
-
- this.partitionColumn = properties.getProperty(PARTITION_COLUMN);
- this.lookback = Period.parse(properties.getProperty(LOOKBACK));
- this.formatter =
DateTimeFormat.forPattern(properties.getProperty(DATETIME_FORMAT));
+ this.prop = (properties == null) ? System.getProperties(): properties;
}
@Override
public String getFilter(HiveDataset hiveDataset) {
- DateTime limitDate = (new DateTime()).minus(this.lookback);
+ if (isValidConfig()) {
Review comment:
Sounds good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 394601)
Time Spent: 1h 40m (was: 1.5h)
> Refactor HiveSource to make partition filter extensible
> -------------------------------------------------------
>
> Key: GOBBLIN-1054
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1054
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Lei Sun
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
