[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[ https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=392349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392349 ] ASF GitHub Bot logged work on GOBBLIN-1054: --- Author: ASF GitHub Bot Created on: 25/Feb/20 07:48 Start Date: 25/Feb/20 07:48 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#issuecomment-587259006 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=h1) Report > Merging [#2894](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/166eb725c01c279068aaefa2c9af93c907bd9eff?src=pr=desc) will **decrease** coverage by `0.04%`. > The diff coverage is `67.5%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2894 +/- ## - Coverage 45.81% 45.77% -0.05% - Complexity 9148 9160 +12 Files 1929 1933 +4 Lines 7260272801 +199 Branches 7993 8026 +33 + Hits 3326433323 +59 - Misses3630336415 +112 - Partials 3035 3063 +28 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../management/conversion/hive/source/HiveSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9zb3VyY2UvSGl2ZVNvdXJjZS5qYXZh) | `66.85% <100%> (+3.26%)` | `24 <0> (ø)` | :arrow_down: | | [...hive/filter/DateRangePartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvRGF0ZVJhbmdlUGFydGl0aW9uRmlsdGVyR2VuZXJhdG9yLmphdmE=) | `50% <50%> (ø)` | `4 <4> (?)` | | | [.../hive/filter/LookbackPartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvTG9va2JhY2tQYXJ0aXRpb25GaWx0ZXJHZW5lcmF0b3IuamF2YQ==) | `78.94% <76.47%> (-21.06%)` | `6 <5> (+3)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `82.25% <0%> (-2.42%)` | `28% <0%> (-1%)` | | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
codecov-io edited a comment on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#issuecomment-587259006 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=h1) Report > Merging [#2894](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/166eb725c01c279068aaefa2c9af93c907bd9eff?src=pr=desc) will **decrease** coverage by `0.04%`. > The diff coverage is `67.5%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2894 +/- ## - Coverage 45.81% 45.77% -0.05% - Complexity 9148 9160 +12 Files 1929 1933 +4 Lines 7260272801 +199 Branches 7993 8026 +33 + Hits 3326433323 +59 - Misses3630336415 +112 - Partials 3035 3063 +28 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2894?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../management/conversion/hive/source/HiveSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9zb3VyY2UvSGl2ZVNvdXJjZS5qYXZh) | `66.85% <100%> (+3.26%)` | `24 <0> (ø)` | :arrow_down: | | [...hive/filter/DateRangePartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvRGF0ZVJhbmdlUGFydGl0aW9uRmlsdGVyR2VuZXJhdG9yLmphdmE=) | `50% <50%> (ø)` | `4 <4> (?)` | | | [.../hive/filter/LookbackPartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvTG9va2JhY2tQYXJ0aXRpb25GaWx0ZXJHZW5lcmF0b3IuamF2YQ==) | `78.94% <76.47%> (-21.06%)` | `6 <5> (+3)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `82.25% <0%> (-2.42%)` | `28% <0%> (-1%)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.87% <0%> (-1.68%)` | `10% <0%> (ø)` | | | ... and [23 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2894/diff?src=pr=tree-more) | | -- [Continue to review
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[ https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=392348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392348 ] ASF GitHub Bot logged work on GOBBLIN-1054: --- Author: ASF GitHub Bot Created on: 25/Feb/20 07:46 Start Date: 25/Feb/20 07:46 Worklog Time Spent: 10m Work Description: KumaravelKandasami commented on pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383705102 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java ## @@ -27,6 +27,7 @@ import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang.StringUtils; +import org.apache.gobblin.data.management.copy.hive.PartitionFilterGenerator; Review comment: Fixed. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392348) Time Spent: 1h 20m (was: 1h 10m) > Refactor HiveSource to make partition filter extensible > --- > > Key: GOBBLIN-1054 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1054 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383705102 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java ## @@ -27,6 +27,7 @@ import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang.StringUtils; +import org.apache.gobblin.data.management.copy.hive.PartitionFilterGenerator; Review comment: Fixed. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[
https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=392346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392346
]
ASF GitHub Bot logged work on GOBBLIN-1054:
---
Author: ASF GitHub Bot
Created on: 25/Feb/20 07:41
Start Date: 25/Feb/20 07:41
Worklog Time Spent: 10m
Work Description: KumaravelKandasami commented on pull request #2894:
GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383703426
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(LOOKBACK),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(DATETIME_FORMAT),
ERROR_MESSAGE);
-
-this.partitionColumn = properties.getProperty(PARTITION_COLUMN);
-this.lookback = Period.parse(properties.getProperty(LOOKBACK));
-this.formatter =
DateTimeFormat.forPattern(properties.getProperty(DATETIME_FORMAT));
+this.prop = (properties == null) ? System.getProperties(): properties;
}
@Override
public String getFilter(HiveDataset hiveDataset) {
-DateTime limitDate = (new DateTime()).minus(this.lookback);
+if (isValidConfig()) {
Review comment:
I do agree with you on keeping the code backward compatible, however, the
prior implementation of the `HiveSource.java `checks whether the expected
properties for `LookbackPartitionFilterGenerator` exists before instantiating
the `LookbackPartitionFilterGenerator`, so the `IllegalArgumentException` is
never raised during runtime. Also, the prior implementation of the
`HiveSource.java `support no PartitionFilter case. To support these existing
behavior, we cannot throw exception when the validation fails.
Reference to prior implementation of `HiveSource.java`:
https://github.com/apache/incubator-gobblin/pull/2894/commits/dca6e665020b147c4009f0f23511c228c64415b6#diff-071355ec76a4f7e8167359040ff492f0L306-L313
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392346)
Time Spent: 1h 10m (was: 1h)
> Refactor HiveSource to make partition filter extensible
> ---
>
> Key: GOBBLIN-1054
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1054
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054:
Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383703426
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(LOOKBACK),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(DATETIME_FORMAT),
ERROR_MESSAGE);
-
-this.partitionColumn = properties.getProperty(PARTITION_COLUMN);
-this.lookback = Period.parse(properties.getProperty(LOOKBACK));
-this.formatter =
DateTimeFormat.forPattern(properties.getProperty(DATETIME_FORMAT));
+this.prop = (properties == null) ? System.getProperties(): properties;
}
@Override
public String getFilter(HiveDataset hiveDataset) {
-DateTime limitDate = (new DateTime()).minus(this.lookback);
+if (isValidConfig()) {
Review comment:
I do agree with you on keeping the code backward compatible, however, the
prior implementation of the `HiveSource.java `checks whether the expected
properties for `LookbackPartitionFilterGenerator` exists before instantiating
the `LookbackPartitionFilterGenerator`, so the `IllegalArgumentException` is
never raised during runtime. Also, the prior implementation of the
`HiveSource.java `support no PartitionFilter case. To support these existing
behavior, we cannot throw exception when the validation fails.
Reference to prior implementation of `HiveSource.java`:
https://github.com/apache/incubator-gobblin/pull/2894/commits/dca6e665020b147c4009f0f23511c228c64415b6#diff-071355ec76a4f7e8167359040ff492f0L306-L313
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[
https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=392339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392339
]
ASF GitHub Bot logged work on GOBBLIN-1054:
---
Author: ASF GitHub Bot
Created on: 25/Feb/20 07:27
Start Date: 25/Feb/20 07:27
Worklog Time Spent: 10m
Work Description: KumaravelKandasami commented on pull request #2894:
GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383698568
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
Review comment:
Same ERROR_MESSAGE is printed when the validation is false in the
getFilter() method.
log.error(LookbackPartitionFilterGenerator.class.getName()
+ " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT}));
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392339)
Time Spent: 1h (was: 50m)
> Refactor HiveSource to make partition filter extensible
> ---
>
> Key: GOBBLIN-1054
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1054
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
KumaravelKandasami commented on a change in pull request #2894: GOBBLIN-1054:
Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r383698568
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
Review comment:
Same ERROR_MESSAGE is printed when the validation is false in the
getFilter() method.
log.error(LookbackPartitionFilterGenerator.class.getName()
+ " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT}));
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
[
https://issues.apache.org/jira/browse/GOBBLIN-1058?focusedWorklogId=392239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392239
]
ASF GitHub Bot logged work on GOBBLIN-1058:
---
Author: ASF GitHub Bot
Created on: 25/Feb/20 02:05
Start Date: 25/Feb/20 02:05
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2898: [GOBBLIN-1058]
Refactor method emitting GTE for ease of adding new tags
URL: https://github.com/apache/incubator-gobblin/pull/2898#discussion_r383616119
##
File path:
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractorStatsTracker.java
##
@@ -319,18 +319,52 @@ public void updateStatisticsForCurrentPartition(int
partitionIdx, long readStart
*/
public void emitTrackingEvents(MetricContext context, MultiLongWatermark
lowWatermark, MultiLongWatermark highWatermark,
MultiLongWatermark nextWatermark) {
+emitTrackingEventsWithAdditionalTags(context, lowWatermark, highWatermark,
nextWatermark, Maps.newHashMap());
+ }
+
+ /**
+ * Emit Tracking events reporting the various statistics to be consumed by a
monitoring application, with additional
+ * map representing tags beyond what are constructed in {@link
#createTagsForPartition(int, MultiLongWatermark, MultiLongWatermark,
MultiLongWatermark) }
+ *
+ * Choose to not to make createTagsForPartition extensible to avoid
additional derived class just for additional k-v pairs
+ * in the tag maps.
+ *
+ * @param additionalTags caller-provided mapping from {@link KafkaPartition}
to {@link Map}, which will
+ * be merged with result of {@link
#createTagsForPartition}.
+ */
+ public void emitTrackingEventsWithAdditionalTags(MetricContext context,
MultiLongWatermark lowWatermark, MultiLongWatermark highWatermark,
+ MultiLongWatermark nextWatermark, Map> additionalTags) {
+Map> tagsForPartitionsMap =
+generateTagsForPartitions(lowWatermark, highWatermark, nextWatermark,
additionalTags);
+
+for (Map.Entry> eventTags :
tagsForPartitionsMap.entrySet()) {
+ EventSubmitter.Builder eventSubmitterBuilder = new
EventSubmitter.Builder(context, GOBBLIN_KAFKA_NAMESPACE);
+
eventSubmitterBuilder.addMetadata(this.taskEventMetadataGenerator.getMetadata(workUnitState,
KAFKA_EXTRACTOR_TOPIC_METADATA_EVENT_NAME));
+
eventSubmitterBuilder.build().submit(KAFKA_EXTRACTOR_TOPIC_METADATA_EVENT_NAME,
eventTags.getValue());
+}
+ }
+
+ /**
+ * A helper function to merge tags for KafkaPartition. Separate into a
package-private method for ease of testing.
+ */
+ Map>
generateTagsForPartitions(MultiLongWatermark lowWatermark, MultiLongWatermark
highWatermark,
Review comment:
@VisibleforTesting?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392239)
Time Spent: 0.5h (was: 20m)
> Make emitTrackingEvents method accept additional PartitionsToTags map for
> ease of extension on metrics
> --
>
> Key: GOBBLIN-1058
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1058
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags
sv2000 commented on a change in pull request #2898: [GOBBLIN-1058] Refactor
method emitting GTE for ease of adding new tags
URL: https://github.com/apache/incubator-gobblin/pull/2898#discussion_r383616119
##
File path:
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractorStatsTracker.java
##
@@ -319,18 +319,52 @@ public void updateStatisticsForCurrentPartition(int
partitionIdx, long readStart
*/
public void emitTrackingEvents(MetricContext context, MultiLongWatermark
lowWatermark, MultiLongWatermark highWatermark,
MultiLongWatermark nextWatermark) {
+emitTrackingEventsWithAdditionalTags(context, lowWatermark, highWatermark,
nextWatermark, Maps.newHashMap());
+ }
+
+ /**
+ * Emit Tracking events reporting the various statistics to be consumed by a
monitoring application, with additional
+ * map representing tags beyond what are constructed in {@link
#createTagsForPartition(int, MultiLongWatermark, MultiLongWatermark,
MultiLongWatermark) }
+ *
+ * Choose to not to make createTagsForPartition extensible to avoid
additional derived class just for additional k-v pairs
+ * in the tag maps.
+ *
+ * @param additionalTags caller-provided mapping from {@link KafkaPartition}
to {@link Map}, which will
+ * be merged with result of {@link
#createTagsForPartition}.
+ */
+ public void emitTrackingEventsWithAdditionalTags(MetricContext context,
MultiLongWatermark lowWatermark, MultiLongWatermark highWatermark,
+ MultiLongWatermark nextWatermark, Map> additionalTags) {
+Map> tagsForPartitionsMap =
+generateTagsForPartitions(lowWatermark, highWatermark, nextWatermark,
additionalTags);
+
+for (Map.Entry> eventTags :
tagsForPartitionsMap.entrySet()) {
+ EventSubmitter.Builder eventSubmitterBuilder = new
EventSubmitter.Builder(context, GOBBLIN_KAFKA_NAMESPACE);
+
eventSubmitterBuilder.addMetadata(this.taskEventMetadataGenerator.getMetadata(workUnitState,
KAFKA_EXTRACTOR_TOPIC_METADATA_EVENT_NAME));
+
eventSubmitterBuilder.build().submit(KAFKA_EXTRACTOR_TOPIC_METADATA_EVENT_NAME,
eventTags.getValue());
+}
+ }
+
+ /**
+ * A helper function to merge tags for KafkaPartition. Separate into a
package-private method for ease of testing.
+ */
+ Map>
generateTagsForPartitions(MultiLongWatermark lowWatermark, MultiLongWatermark
highWatermark,
Review comment:
@VisibleforTesting?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-1057. Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request #2897 [https://github.com/apache/incubator-gobblin/pull/2897] > Remove unnecessary RPCs in distcp-ng > > > Key: GOBBLIN-1057 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h > Remaining Estimate: 0h > > There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. > This results in a long file discovery phase that can be hours for a few > thousand files. > The RPCs that can be removed are: > getFileChecksum() - the value doesn't appear to be used. > getFileStatus() - this is called to get the modification time in > ModTimeDataFileVersionStrategy.getVersion(). The modification time is already > available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392199 ] ASF GitHub Bot logged work on GOBBLIN-1057: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:51 Start Date: 24/Feb/20 23:51 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392199) Time Spent: 1h (was: 50m) > Remove unnecessary RPCs in distcp-ng > > > Key: GOBBLIN-1057 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h > Remaining Estimate: 0h > > There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. > This results in a long file discovery phase that can be hours for a few > thousand files. > The RPCs that can be removed are: > getFileChecksum() - the value doesn't appear to be used. > getFileStatus() - this is called to get the modification time in > ModTimeDataFileVersionStrategy.getVersion(). The modification time is already > available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
asfgit closed pull request #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
[ https://issues.apache.org/jira/browse/GOBBLIN-1058?focusedWorklogId=392182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392182 ] ASF GitHub Bot logged work on GOBBLIN-1058: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:30 Start Date: 24/Feb/20 23:30 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **increase** coverage by `0.02%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2898 +/- ## + Coverage 45.84% 45.86% +0.02% - Complexity 9151 9168 +17 Files 1929 1932 +3 Lines 7264972778 +129 Branches 7999 8014 +15 + Hits 3330433382 +78 - Misses3630536348 +43 - Partials 3040 3048 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `85.97% <28.57%> (+31.48%)` | `29 <0> (+4)` | :arrow_up: | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `41.73% <0%> (-0.74%)` | `15% <0%> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.28% <0%> (-0.72%)` | `27%
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags
codecov-io edited a comment on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **increase** coverage by `0.02%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2898 +/- ## + Coverage 45.84% 45.86% +0.02% - Complexity 9151 9168 +17 Files 1929 1932 +3 Lines 7264972778 +129 Branches 7999 8014 +15 + Hits 3330433382 +78 - Misses3630536348 +43 - Partials 3040 3048 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `85.97% <28.57%> (+31.48%)` | `29 <0> (+4)` | :arrow_up: | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `41.73% <0%> (-0.74%)` | `15% <0%> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.28% <0%> (-0.72%)` | `27% <0%> (ø)` | | | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `21.18% <0%> (-0.22%)` | `8% <0%> (ø)` | | | ... and [14 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree-more) | |
[jira] [Work logged] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
[ https://issues.apache.org/jira/browse/GOBBLIN-1058?focusedWorklogId=392181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392181 ] ASF GitHub Bot logged work on GOBBLIN-1058: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:28 Start Date: 24/Feb/20 23:28 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **decrease** coverage by `41.71%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2898 +/- ## - Coverage 45.84% 4.12% -41.72% + Complexity 9151 750 -8401 Files 19291932+3 Lines 72649 72778 +129 Branches 79998014 +15 - Hits 333043002-30302 - Misses36305 69458+33153 + Partials 3040 318 -2722 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `0% <0%> (-54.49%)` | `0 <0> (-25)` | | | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%>
[GitHub] [incubator-gobblin] codecov-io commented on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags
codecov-io commented on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **decrease** coverage by `41.71%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2898 +/- ## - Coverage 45.84% 4.12% -41.72% + Complexity 9151 750 -8401 Files 19291932+3 Lines 72649 72778 +129 Branches 79998014 +15 - Hits 333043002-30302 - Misses36305 69458+33153 + Partials 3040 318 -2722 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `0% <0%> (-54.49%)` | `0 <0> (-25)` | | | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...bblin/data/management/copy/AllEqualComparator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQWxsRXF1YWxDb21wYXJhdG9yLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | ... and [1117
[jira] [Work logged] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392174 ] ASF GitHub Bot logged work on GOBBLIN-1057: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:19 Start Date: 24/Feb/20 23:19 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590596708 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=h1) Report > Merging [#2897](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/096fc0c78f820df26958f22b53558ca4b082f7ce?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `38.46%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2897 +/- ## + Coverage 45.77% 45.77% +<.01% - Complexity 9152 9156 +4 Files 1932 1932 Lines 7276872777 +9 Branches 8013 8021 +8 + Hits 3330733313 +6 + Misses3641836413 -5 - Partials 3043 3051 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...che/gobblin/data/management/copy/CopyableFile.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weWFibGVGaWxlLmphdmE=) | `81.39% <100%> (+0.21%)` | `33 <1> (+1)` | :arrow_up: | | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `22.6% <33.33%> (-0.27%)` | `14 <0> (ø)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.66% <33.33%> (-1.89%)` | `10 <0> (ø)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.2% <0%> (-0.68%)` | `24% <0%> (-1%)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.5% <0%> (+0.43%)` | `29% <0%> (ø)` | :arrow_down: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [...obblin/data/management/copy/CopyConfiguration.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weUNvbmZpZ3VyYXRpb24uamF2YQ==) | `85% <0%> (+5%)` | `14% <0%> (+1%)` | :arrow_up: | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
codecov-io edited a comment on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590596708 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=h1) Report > Merging [#2897](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/096fc0c78f820df26958f22b53558ca4b082f7ce?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `38.46%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2897 +/- ## + Coverage 45.77% 45.77% +<.01% - Complexity 9152 9156 +4 Files 1932 1932 Lines 7276872777 +9 Branches 8013 8021 +8 + Hits 3330733313 +6 + Misses3641836413 -5 - Partials 3043 3051 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...che/gobblin/data/management/copy/CopyableFile.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weWFibGVGaWxlLmphdmE=) | `81.39% <100%> (+0.21%)` | `33 <1> (+1)` | :arrow_up: | | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `22.6% <33.33%> (-0.27%)` | `14 <0> (ø)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.66% <33.33%> (-1.89%)` | `10 <0> (ø)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.2% <0%> (-0.68%)` | `24% <0%> (-1%)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.5% <0%> (+0.43%)` | `29% <0%> (ø)` | :arrow_down: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [...obblin/data/management/copy/CopyConfiguration.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weUNvbmZpZ3VyYXRpb24uamF2YQ==) | `85% <0%> (+5%)` | `14% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `41.73% <0%> (+6.08%)` | `15% <0%> (+2%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=continue). >
[jira] [Work logged] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392171 ] ASF GitHub Bot logged work on GOBBLIN-1057: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:14 Start Date: 24/Feb/20 23:14 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590596708 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=h1) Report > Merging [#2897](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/096fc0c78f820df26958f22b53558ca4b082f7ce?src=pr=desc) will **decrease** coverage by `41.67%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2897 +/- ## - Coverage 45.77%4.1% -41.68% + Complexity 9152 749 -8403 Files 19321932 Lines 72768 72777+9 Branches 80138021+8 - Hits 333072984-30323 - Misses36418 69475+33057 + Partials 3043 318 -2725 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...che/gobblin/data/management/copy/CopyableFile.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weWFibGVGaWxlLmphdmE=) | `0% <0%> (-81.18%)` | `0 <0> (-32)` | | | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0% <0%> (-22.88%)` | `0 <0> (-14)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `0% <0%> (-70.55%)` | `0 <0> (-10)` | | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | |
[GitHub] [incubator-gobblin] codecov-io commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
codecov-io commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590596708 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=h1) Report > Merging [#2897](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/096fc0c78f820df26958f22b53558ca4b082f7ce?src=pr=desc) will **decrease** coverage by `41.67%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2897 +/- ## - Coverage 45.77%4.1% -41.68% + Complexity 9152 749 -8403 Files 19321932 Lines 72768 72777+9 Branches 80138021+8 - Hits 333072984-30323 - Misses36418 69475+33057 + Partials 3043 318 -2725 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2897?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...che/gobblin/data/management/copy/CopyableFile.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weWFibGVGaWxlLmphdmE=) | `0% <0%> (-81.18%)` | `0 <0> (-32)` | | | [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0% <0%> (-22.88%)` | `0 <0> (-14)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `0% <0%> (-70.55%)` | `0 <0> (-10)` | | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...bblin/data/management/copy/AllEqualComparator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQWxsRXF1YWxDb21wYXJhdG9yLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | ... and [1114 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2897/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Work logged] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392157 ] ASF GitHub Bot logged work on GOBBLIN-1057: --- Author: ASF GitHub Bot Created on: 24/Feb/20 23:03 Start Date: 24/Feb/20 23:03 Worklog Time Spent: 10m Work Description: yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590593472 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392157) Time Spent: 0.5h (was: 20m) > Remove unnecessary RPCs in distcp-ng > > > Key: GOBBLIN-1057 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Hung Tran >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. > This results in a long file discovery phase that can be hours for a few > thousand files. > The RPCs that can be removed are: > getFileChecksum() - the value doesn't appear to be used. > getFileStatus() - this is called to get the modification time in > ModTimeDataFileVersionStrategy.getVersion(). The modification time is already > available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590593472 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392140 ] ASF GitHub Bot logged work on GOBBLIN-1057: --- Author: ASF GitHub Bot Created on: 24/Feb/20 22:45 Start Date: 24/Feb/20 22:45 Worklog Time Spent: 10m Work Description: yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590587556 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392140) Time Spent: 20m (was: 10m) > Remove unnecessary RPCs in distcp-ng > > > Key: GOBBLIN-1057 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Hung Tran >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. > This results in a long file discovery phase that can be hours for a few > thousand files. > The RPCs that can be removed are: > getFileChecksum() - the value doesn't appear to be used. > getFileStatus() - this is called to get the modification time in > ModTimeDataFileVersionStrategy.getVersion(). The modification time is already > available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
yukuai518 commented on issue #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng URL: https://github.com/apache/incubator-gobblin/pull/2897#issuecomment-590587556 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2898: []Refactor method emitting GTE for ease of adding new tags
autumnust opened a new pull request #2898: []Refactor method emitting GTE for
ease of adding new tags
URL: https://github.com/apache/incubator-gobblin/pull/2898
…nto metrics
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
Lei Sun created GOBBLIN-1058: Summary: Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics Key: GOBBLIN-1058 URL: https://issues.apache.org/jira/browse/GOBBLIN-1058 Project: Apache Gobblin Issue Type: Improvement Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1057) Remove unnecessary RPCs in distcp-ng
[ https://issues.apache.org/jira/browse/GOBBLIN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-1057: --- Summary: Remove unnecessary RPCs in distcp-ng (was: Optimize unnecessary RPCs in distcp-ng) > Remove unnecessary RPCs in distcp-ng > > > Key: GOBBLIN-1057 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Hung Tran >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. > This results in a long file discovery phase that can be hours for a few > thousand files. > The RPCs that can be removed are: > getFileChecksum() - the value doesn't appear to be used. > getFileStatus() - this is called to get the modification time in > ModTimeDataFileVersionStrategy.getVersion(). The modification time is already > available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1057) Optimize unnecessary RPCs in distcp-ng
[
https://issues.apache.org/jira/browse/GOBBLIN-1057?focusedWorklogId=392127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392127
]
ASF GitHub Bot logged work on GOBBLIN-1057:
---
Author: ASF GitHub Bot
Created on: 24/Feb/20 22:28
Start Date: 24/Feb/20 22:28
Worklog Time Spent: 10m
Work Description: htran1 commented on pull request #2897: [GOBBLIN-1057]
Remove unnecessary RPCs in distcp-ng
URL: https://github.com/apache/incubator-gobblin/pull/2897
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [X] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1057
### Description
- [X] Here are some details about my PR, including screenshots (if
applicable):
Disable fetching of the source checksum by default.
There is no code using it, so this should not have any impact.
If the DataFileVersionStrategy is ModTimeDataFileVersionStrategy then
optimize the code path by using the modification time in the FileStatus
object that has already been fetched.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Ran existing tests.
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392127)
Remaining Estimate: 0h
Time Spent: 10m
> Optimize unnecessary RPCs in distcp-ng
> --
>
> Key: GOBBLIN-1057
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1057
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Hung Tran
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng.
> This results in a long file discovery phase that can be hours for a few
> thousand files.
> The RPCs that can be removed are:
> getFileChecksum() - the value doesn't appear to be used.
> getFileStatus() - this is called to get the modification time in
> ModTimeDataFileVersionStrategy.getVersion(). The modification time is already
> available from listStatus(), so use that value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] htran1 opened a new pull request #2897: [GOBBLIN-1057] Remove unnecessary RPCs in distcp-ng
htran1 opened a new pull request #2897: [GOBBLIN-1057] Remove unnecessary RPCs
in distcp-ng
URL: https://github.com/apache/incubator-gobblin/pull/2897
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [X] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1057
### Description
- [X] Here are some details about my PR, including screenshots (if
applicable):
Disable fetching of the source checksum by default.
There is no code using it, so this should not have any impact.
If the DataFileVersionStrategy is ModTimeDataFileVersionStrategy then
optimize the code path by using the modification time in the FileStatus
object that has already been fetched.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Ran existing tests.
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-1057) Optimize unnecessary RPCs in distcp-ng
Hung Tran created GOBBLIN-1057: -- Summary: Optimize unnecessary RPCs in distcp-ng Key: GOBBLIN-1057 URL: https://issues.apache.org/jira/browse/GOBBLIN-1057 Project: Apache Gobblin Issue Type: Improvement Reporter: Hung Tran There are some per-file FileSystem RPCs being invoked in Gobblin distcp-ng. This results in a long file discovery phase that can be hours for a few thousand files. The RPCs that can be removed are: getFileChecksum() - the value doesn't appear to be used. getFileStatus() - this is called to get the modification time in ModTimeDataFileVersionStrategy.getVersion(). The modification time is already available from listStatus(), so use that value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1052) Create a spec consumer path if it does not exist in FS SpecConsumer
[ https://issues.apache.org/jira/browse/GOBBLIN-1052?focusedWorklogId=392038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392038 ] ASF GitHub Bot logged work on GOBBLIN-1052: --- Author: ASF GitHub Bot Created on: 24/Feb/20 19:53 Start Date: 24/Feb/20 19:53 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2892: GOBBLIN-1052: Create a spec consumer path if it does not exist in FS … URL: https://github.com/apache/incubator-gobblin/pull/2892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392038) Time Spent: 1h 20m (was: 1h 10m) > Create a spec consumer path if it does not exist in FS SpecConsumer > --- > > Key: GOBBLIN-1052 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1052 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Current implementation expects the spec consumer path to already exist. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2892: GOBBLIN-1052: Create a spec consumer path if it does not exist in FS …
asfgit closed pull request #2892: GOBBLIN-1052: Create a spec consumer path if it does not exist in FS … URL: https://github.com/apache/incubator-gobblin/pull/2892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1050) Verify requester when updating/deleting FlowConfig
[ https://issues.apache.org/jira/browse/GOBBLIN-1050?focusedWorklogId=391913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391913 ] ASF GitHub Bot logged work on GOBBLIN-1050: --- Author: ASF GitHub Bot Created on: 24/Feb/20 19:04 Start Date: 24/Feb/20 19:04 Worklog Time Spent: 10m Work Description: jack-moseley commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-590495950 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391913) Time Spent: 0.5h (was: 20m) > Verify requester when updating/deleting FlowConfig > -- > > Key: GOBBLIN-1050 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1050 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jack-moseley commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
jack-moseley commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-590495950 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
