[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[ https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309267 ] ASF GitHub Bot logged work on GOBBLIN-874: -- Author: ASF GitHub Bot Created on: 09/Sep/19 21:32 Start Date: 09/Sep/19 21:32 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2728: [GOBBLIN-874] Make WorkUnitPacker and SizeEstimator pluggable URL: https://github.com/apache/incubator-gobblin/pull/2728 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309267) Time Spent: 1h (was: 50m) > Make KafkaWUPacker and SizeEstimator pluggable > --- > > Key: GOBBLIN-874 > URL: https://issues.apache.org/jira/browse/GOBBLIN-874 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[ https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309137 ] ASF GitHub Bot logged work on GOBBLIN-874: -- Author: ASF GitHub Bot Created on: 09/Sep/19 18:18 Start Date: 09/Sep/19 18:18 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2728: [GOBBLIN-874] Make WorkUnitPacker and SizeEstimator pluggable URL: https://github.com/apache/incubator-gobblin/pull/2728#issuecomment-529603187 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2728?src=pr=h1) Report > Merging [#2728](https://codecov.io/gh/apache/incubator-gobblin/pull/2728?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc) will **increase** coverage by `0.06%`. > The diff coverage is `90.9%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2728?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2728 +/- ## + Coverage 44.99% 45.05% +0.06% - Complexity 8742 8753 +11 Files 1884 1884 Lines 7029570302 +7 Branches 7715 7715 + Hits 3162931675 +46 + Misses3573535688 -47 - Partials 2931 2939 +8 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2728?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...orkunit/packer/KafkaSingleLevelWorkUnitPacker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FTaW5nbGVMZXZlbFdvcmtVbml0UGFja2VyLmphdmE=) | `12.5% <ø> (+12.5%)` | `1 <0> (+1)` | :arrow_up: | | [...ka/workunit/packer/KafkaBiLevelWorkUnitPacker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FCaUxldmVsV29ya1VuaXRQYWNrZXIuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../KafkaAvgRecordSizeBasedWorkUnitSizeEstimator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FBdmdSZWNvcmRTaXplQmFzZWRXb3JrVW5pdFNpemVFc3RpbWF0b3IuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...act/kafka/workunit/packer/KafkaWorkUnitPacker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FXb3JrVW5pdFBhY2tlci5qYXZh) | `16.48% <100%> (+16.48%)` | `6 <0> (+6)` | :arrow_up: | | [.../KafkaAvgRecordTimeBasedWorkUnitSizeEstimator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FBdmdSZWNvcmRUaW1lQmFzZWRXb3JrVW5pdFNpemVFc3RpbWF0b3IuamF2YQ==) | `26.08% <100%> (+26.08%)` | `3 <0> (+3)` | :arrow_up: | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2728/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `66.19%
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[
https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309064
]
ASF GitHub Bot logged work on GOBBLIN-874:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 16:47
Start Date: 09/Sep/19 16:47
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2728: [GOBBLIN-874]
Make WorkUnitPacker and SizeEstimator pluggable
URL: https://github.com/apache/incubator-gobblin/pull/2728#discussion_r322343871
##
File path:
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPacker.java
##
@@ -58,20 +59,34 @@
private static final Logger LOG =
LoggerFactory.getLogger(KafkaWorkUnitPacker.class);
+ /**
+ * For customized type of the following enums, it will try to find declared
class in classpath
+ * and fallback to exception if ClassNotFound. This way the sizeEstimator
and packer could be easier to
+ * extend. The major purpose for keeping this enum instead of using
reflection only to construct the instance
+ * of packer or sizeEstimator is to maintain backward-compatibility.
+ *
+ * The constructor of customized type needs to be annotated with public
access-modifier as it is instantiated by
+ * {@link GobblinConstructorUtils} which reside in different package, and it
needs to have the same signature
+ * as other implementation under the enum.
+ */
public enum PackerType {
SINGLE_LEVEL,
-BI_LEVEL
+BI_LEVEL,
+CUSTOMIZED
Review comment:
CUSTOM instead of CUSTOMIZED?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309064)
Time Spent: 0.5h (was: 20m)
> Make KafkaWUPacker and SizeEstimator pluggable
> ---
>
> Key: GOBBLIN-874
> URL: https://issues.apache.org/jira/browse/GOBBLIN-874
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[
https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309065
]
ASF GitHub Bot logged work on GOBBLIN-874:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 16:47
Start Date: 09/Sep/19 16:47
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2728: [GOBBLIN-874]
Make WorkUnitPacker and SizeEstimator pluggable
URL: https://github.com/apache/incubator-gobblin/pull/2728#discussion_r322345045
##
File path:
gobblin-modules/gobblin-kafka-common/src/test/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPackerTest.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.source.extractor.extract.kafka.workunit.packer;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.gobblin.configuration.SourceState;
+import org.apache.gobblin.source.extractor.extract.AbstractSource;
+import org.apache.gobblin.source.workunit.WorkUnit;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static
org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.KAFKA_WORKUNIT_PACKER_CUSTOMIZED_TYPE;
+import static
org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.KAFKA_WORKUNIT_PACKER_TYPE;
+import static
org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.KAFKA_WORKUNIT_SIZE_ESTIMATOR_CUSTOMIZED_TYPE;
+import static
org.apache.gobblin.source.extractor.extract.kafka.workunit.packer.KafkaWorkUnitPacker.KAFKA_WORKUNIT_SIZE_ESTIMATOR_TYPE;
+
+
+public class KafkaWorkUnitPackerTest {
+ private KafkaWorkUnitPacker packer;
+ AbstractSource source = Mockito.mock(AbstractSource.class);
+ SourceState state;
+
+ @BeforeMethod
+ public void setUp() {
+state = new SourceState();
+
+// Using customized type and having customized as a known class.
+state.setProp(KAFKA_WORKUNIT_PACKER_TYPE, "CUSTOMIZED");
Review comment:
CUSTOM?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309065)
Time Spent: 40m (was: 0.5h)
> Make KafkaWUPacker and SizeEstimator pluggable
> ---
>
> Key: GOBBLIN-874
> URL: https://issues.apache.org/jira/browse/GOBBLIN-874
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[ https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309062 ] ASF GitHub Bot logged work on GOBBLIN-874: -- Author: ASF GitHub Bot Created on: 09/Sep/19 16:47 Start Date: 09/Sep/19 16:47 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2728: [GOBBLIN-874] Make WorkUnitPacker and SizeEstimator pluggable URL: https://github.com/apache/incubator-gobblin/pull/2728#discussion_r322343951 ## File path: gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPacker.java ## @@ -357,6 +377,10 @@ public static KafkaWorkUnitPacker getInstance(PackerType packerType, AbstractSou return new KafkaSingleLevelWorkUnitPacker(source, state); case BI_LEVEL: return new KafkaBiLevelWorkUnitPacker(source, state); + case CUSTOMIZED: Review comment: Same comment as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309062) Time Spent: 20m (was: 10m) > Make KafkaWUPacker and SizeEstimator pluggable > --- > > Key: GOBBLIN-874 > URL: https://issues.apache.org/jira/browse/GOBBLIN-874 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[ https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309063 ] ASF GitHub Bot logged work on GOBBLIN-874: -- Author: ASF GitHub Bot Created on: 09/Sep/19 16:47 Start Date: 09/Sep/19 16:47 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2728: [GOBBLIN-874] Make WorkUnitPacker and SizeEstimator pluggable URL: https://github.com/apache/incubator-gobblin/pull/2728#discussion_r322344949 ## File path: gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/workunit/packer/KafkaWorkUnitPacker.java ## @@ -58,20 +59,34 @@ private static final Logger LOG = LoggerFactory.getLogger(KafkaWorkUnitPacker.class); + /** + * For customized type of the following enums, it will try to find declared class in classpath + * and fallback to exception if ClassNotFound. This way the sizeEstimator and packer could be easier to + * extend. The major purpose for keeping this enum instead of using reflection only to construct the instance Review comment: +1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309063) Time Spent: 0.5h (was: 20m) > Make KafkaWUPacker and SizeEstimator pluggable > --- > > Key: GOBBLIN-874 > URL: https://issues.apache.org/jira/browse/GOBBLIN-874 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-874) Make KafkaWUPacker and SizeEstimator pluggable
[
https://issues.apache.org/jira/browse/GOBBLIN-874?focusedWorklogId=309052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309052
]
ASF GitHub Bot logged work on GOBBLIN-874:
--
Author: ASF GitHub Bot
Created on: 09/Sep/19 16:37
Start Date: 09/Sep/19 16:37
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2728:
[GOBBLIN-874] Make WorkUnitPacker and SizeEstimator pluggable
URL: https://github.com/apache/incubator-gobblin/pull/2728
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following
- https://issues.apache.org/jira/browse/GOBBLIN-874
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Leave the original enum there for backward-compatible.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309052)
Remaining Estimate: 0h
Time Spent: 10m
> Make KafkaWUPacker and SizeEstimator pluggable
> ---
>
> Key: GOBBLIN-874
> URL: https://issues.apache.org/jira/browse/GOBBLIN-874
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
