[jira] [Work logged] (GOBBLIN-1199) convert seconds to ms because helix api take time in ms
[ https://issues.apache.org/jira/browse/GOBBLIN-1199?focusedWorklogId=446920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446920 ] ASF GitHub Bot logged work on GOBBLIN-1199: --- Author: ASF GitHub Bot Created on: 16/Jun/20 21:23 Start Date: 16/Jun/20 21:23 Worklog Time Spent: 10m Work Description: codecov-commenter commented on pull request #3045: URL: https://github.com/apache/incubator-gobblin/pull/3045#issuecomment-645019159 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=h1) Report > Merging [#3045](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.03%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3045 +/- ## + Coverage 45.78% 45.81% +0.03% - Complexity 9328 9358 +30 Files 1956 1962 +6 Lines 7442374558 +135 Branches 8250 8264 +14 + Hits 3407334160 +87 - Misses3715737207 +50 + Partials 3193 3191 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...er/GobblinHelixDistributeJobExecutionLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4RGlzdHJpYnV0ZUpvYkV4ZWN1dGlvbkxhdW5jaGVyLmphdmE=) | `57.93% <0.00%> (ø)` | `10.00 <0.00> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `78.81% <0.00%> (ø)` | `22.00 <0.00> (ø)` | | | [...blin/runtime/services/MetricsReportingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc2VydmljZXMvTWV0cmljc1JlcG9ydGluZ1NlcnZpY2UuamF2YQ==) | `80.00% <0.00%> (-20.00%)` | `4.00% <0.00%> (+1.00%)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `80.28% <0.00%> (-2.08%)` | `12.00% <0.00%> (ø%)` | | | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `62.79% <0.00%> (-1.50%)` | `6.00% <0.00%> (ø%)` | | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0.00%> (-1.43%)` | `4.00% <0.00%> (ø%)` | | | [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `51.93% <0.00%> (-1.12%)` | `19.00% <0.00%> (ø%)` | | | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `14.54% <0.00%> (-0.20%)` | `4.00% <0.00%> (ø%)` | | | [...ache/gobblin/compaction/mapreduce/MRCompactor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL01SQ29tcGFjdG9yLmphdmE=) | `7.04% <0.00%> (-0.10%)` | `9.00% <0.00%> (ø%)` | | |
[GitHub] [incubator-gobblin] codecov-commenter commented on pull request #3045: [GOBBLIN-1199] convert seconds to ms because helix api take time in ms
codecov-commenter commented on pull request #3045: URL: https://github.com/apache/incubator-gobblin/pull/3045#issuecomment-645019159 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=h1) Report > Merging [#3045](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.03%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3045 +/- ## + Coverage 45.78% 45.81% +0.03% - Complexity 9328 9358 +30 Files 1956 1962 +6 Lines 7442374558 +135 Branches 8250 8264 +14 + Hits 3407334160 +87 - Misses3715737207 +50 + Partials 3193 3191 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...er/GobblinHelixDistributeJobExecutionLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4RGlzdHJpYnV0ZUpvYkV4ZWN1dGlvbkxhdW5jaGVyLmphdmE=) | `57.93% <0.00%> (ø)` | `10.00 <0.00> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `78.81% <0.00%> (ø)` | `22.00 <0.00> (ø)` | | | [...blin/runtime/services/MetricsReportingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc2VydmljZXMvTWV0cmljc1JlcG9ydGluZ1NlcnZpY2UuamF2YQ==) | `80.00% <0.00%> (-20.00%)` | `4.00% <0.00%> (+1.00%)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `80.28% <0.00%> (-2.08%)` | `12.00% <0.00%> (ø%)` | | | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `62.79% <0.00%> (-1.50%)` | `6.00% <0.00%> (ø%)` | | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0.00%> (-1.43%)` | `4.00% <0.00%> (ø%)` | | | [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `51.93% <0.00%> (-1.12%)` | `19.00% <0.00%> (ø%)` | | | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `14.54% <0.00%> (-0.20%)` | `4.00% <0.00%> (ø%)` | | | [...ache/gobblin/compaction/mapreduce/MRCompactor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL01SQ29tcGFjdG9yLmphdmE=) | `7.04% <0.00%> (-0.10%)` | `9.00% <0.00%> (ø%)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `2.70% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | | | ... and [19 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3045/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3045?src=pr=continue). > **Legend** - [Click here to learn
[jira] [Work logged] (GOBBLIN-1199) convert seconds to ms because helix api take time in ms
[
https://issues.apache.org/jira/browse/GOBBLIN-1199?focusedWorklogId=446898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446898
]
ASF GitHub Bot logged work on GOBBLIN-1199:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:46
Start Date: 16/Jun/20 20:46
Worklog Time Spent: 10m
Work Description: arjun4084346 opened a new pull request #3045:
URL: https://github.com/apache/incubator-gobblin/pull/3045
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1199
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
convert seconds to ms because helix api take time in ms
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial changes
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446898)
Time Spent: 20m (was: 10m)
> convert seconds to ms because helix api take time in ms
> ---
>
> Key: GOBBLIN-1199
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1199
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1199) convert seconds to ms because helix api take time in ms
[ https://issues.apache.org/jira/browse/GOBBLIN-1199?focusedWorklogId=446897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446897 ] ASF GitHub Bot logged work on GOBBLIN-1199: --- Author: ASF GitHub Bot Created on: 16/Jun/20 20:46 Start Date: 16/Jun/20 20:46 Worklog Time Spent: 10m Work Description: arjun4084346 closed pull request #3045: URL: https://github.com/apache/incubator-gobblin/pull/3045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446897) Remaining Estimate: 0h Time Spent: 10m > convert seconds to ms because helix api take time in ms > --- > > Key: GOBBLIN-1199 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1199 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 closed pull request #3045: [GOBBLIN-1199] convert seconds to ms because helix api take time in ms
arjun4084346 closed pull request #3045: URL: https://github.com/apache/incubator-gobblin/pull/3045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (GOBBLIN-1199) convert seconds to ms because helix api take time in ms
Arjun Singh Bora created GOBBLIN-1199: - Summary: convert seconds to ms because helix api take time in ms Key: GOBBLIN-1199 URL: https://issues.apache.org/jira/browse/GOBBLIN-1199 Project: Apache Gobblin Issue Type: Bug Reporter: Arjun Singh Bora -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446890
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:40
Start Date: 16/Jun/20 20:40
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441128943
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
+this.newSpecs.inc();
+this.jobCatalog.put(parsedMessage);
+break;
+ case DELETE:
+this.removedSpecs.inc();
+URI jobSpecUri = parsedMessage.getUri();
+this.jobCatalog.remove(jobSpecUri);
+// Delete the job state if it is a delete spec request
+deleteStateStore(jobSpecUri);
+break;
+ case CANCEL:
+this.jobCatalog.cancel(parsedMessage);
+break;
+ default:
Review comment:
If we throw an error, job monitor thread will die and then we will not
be able to process other messages.
Really not sure which approach is better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446890)
Time Spent: 2h 10m (was: 2h)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
arjun4084346 commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441128943
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
+this.newSpecs.inc();
+this.jobCatalog.put(parsedMessage);
+break;
+ case DELETE:
+this.removedSpecs.inc();
+URI jobSpecUri = parsedMessage.getUri();
+this.jobCatalog.remove(jobSpecUri);
+// Delete the job state if it is a delete spec request
+deleteStateStore(jobSpecUri);
+break;
+ case CANCEL:
+this.jobCatalog.cancel(parsedMessage);
+break;
+ default:
Review comment:
If we throw an error, job monitor thread will die and then we will not
be able to process other messages.
Really not sure which approach is better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446870
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:19
Start Date: 16/Jun/20 20:19
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441117149
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/AvroJobSpecKafkaJobMonitor.java
##
@@ -127,17 +121,10 @@ protected void createMetrics() {
}
}
-String verbName = record.getMetadata().get(VERB_KEY);
Review comment:
VERB_KEY should be guaranteed, otherwise this will throw exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446870)
Time Spent: 1h 50m (was: 1h 40m)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
arjun4084346 commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441117149
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/AvroJobSpecKafkaJobMonitor.java
##
@@ -127,17 +121,10 @@ protected void createMetrics() {
}
}
-String verbName = record.getMetadata().get(VERB_KEY);
Review comment:
VERB_KEY should be guaranteed, otherwise this would have thrown exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446872
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:19
Start Date: 16/Jun/20 20:19
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441117149
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/AvroJobSpecKafkaJobMonitor.java
##
@@ -127,17 +121,10 @@ protected void createMetrics() {
}
}
-String verbName = record.getMetadata().get(VERB_KEY);
Review comment:
VERB_KEY should be guaranteed, otherwise this would have thrown exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446872)
Time Spent: 2h (was: 1h 50m)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
arjun4084346 commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441117149
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/AvroJobSpecKafkaJobMonitor.java
##
@@ -127,17 +121,10 @@ protected void createMetrics() {
}
}
-String verbName = record.getMetadata().get(VERB_KEY);
Review comment:
VERB_KEY should be guaranteed, otherwise this will throw exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446867
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:15
Start Date: 16/Jun/20 20:15
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441115099
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
Review comment:
yes, verb is guaranteed to be set. unknown is its default value
https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/JobSpec.java#L347
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446867)
Time Spent: 1h 40m (was: 1.5h)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
arjun4084346 commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441115099
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
Review comment:
yes, verb is guaranteed to be set. unknown is its default value
https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/JobSpec.java#L347
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
arjun4084346 commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441110026
##
File path: gobblin-modules/gobblin-kafka-09/build.gradle
##
@@ -87,6 +87,9 @@ artifacts {
}
test {
+ // this module's test cases hang often when running in parallel.
+ // making every test class execute in a forked test process till tests are
fixed
+ forkEvery = 1
Review comment:
yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446864
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:04
Start Date: 16/Jun/20 20:04
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441110026
##
File path: gobblin-modules/gobblin-kafka-09/build.gradle
##
@@ -87,6 +87,9 @@ artifacts {
}
test {
+ // this module's test cases hang often when running in parallel.
+ // making every test class execute in a forked test process till tests are
fixed
+ forkEvery = 1
Review comment:
yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446864)
Time Spent: 1.5h (was: 1h 20m)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1187) cancel a gaas flow when the dag manager is disabled
[
https://issues.apache.org/jira/browse/GOBBLIN-1187?focusedWorklogId=446863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446863
]
ASF GitHub Bot logged work on GOBBLIN-1187:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 20:03
Start Date: 16/Jun/20 20:03
Worklog Time Spent: 10m
Work Description: autumnust commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441108741
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
Review comment:
Shouldn't the default block be the place to maintain compatibility? Is
the VERB guaranteed to be set previously ?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
+this.newSpecs.inc();
+this.jobCatalog.put(parsedMessage);
+break;
+ case DELETE:
+this.removedSpecs.inc();
+URI jobSpecUri = parsedMessage.getUri();
+this.jobCatalog.remove(jobSpecUri);
+// Delete the job state if it is a delete spec request
+deleteStateStore(jobSpecUri);
+break;
+ case CANCEL:
+this.jobCatalog.cancel(parsedMessage);
+break;
+ default:
Review comment:
Shall we throw exceptions here instead of just printing log ?
##
File path: gobblin-modules/gobblin-kafka-09/build.gradle
##
@@ -87,6 +87,9 @@ artifacts {
}
test {
+ // this module's test cases hang often when running in parallel.
+ // making every test class execute in a forked test process till tests are
fixed
+ forkEvery = 1
Review comment:
Q: Is this the only change targeting for the travis issue ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446863)
Time Spent: 1h 20m (was: 1h 10m)
> cancel a gaas flow when the dag manager is disabled
> ---
>
> Key: GOBBLIN-1187
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1187
> Project: Apache Gobblin
> Issue Type: Improvement
>
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #3027: [GOBBLIN-1187] cancel flow execution when dag manager is disabled
autumnust commented on a change in pull request #3027:
URL: https://github.com/apache/incubator-gobblin/pull/3027#discussion_r441108741
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
Review comment:
Shouldn't the default block be the place to maintain compatibility? Is
the VERB guaranteed to be set previously ?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_monitor/KafkaJobMonitor.java
##
@@ -100,17 +101,29 @@ protected void shutdownMetrics()
@Override
protected void processMessage(DecodeableKafkaRecord message) {
try {
- Collection> parsedCollection =
parseJobSpec(message.getValue());
- for (Either parsedMessage : parsedCollection) {
-if (parsedMessage instanceof Either.Left) {
- this.newSpecs.inc();
- this.jobCatalog.put(((Either.Left)
parsedMessage).getLeft());
-} else if (parsedMessage instanceof Either.Right) {
- this.removedSpecs.inc();
- URI jobSpecUri = ((Either.Right)
parsedMessage).getRight();
- this.jobCatalog.remove(jobSpecUri);
- // Delete the job state if it is a delete spec request
- deleteStateStore(jobSpecUri);
+ Collection parsedCollection = parseJobSpec(message.getValue());
+ for (JobSpec parsedMessage : parsedCollection) {
+SpecExecutor.Verb verb =
SpecExecutor.Verb.valueOf(parsedMessage.getMetadata().get(VERB_KEY));
+
+switch (verb) {
+ case ADD:
+ case UPDATE:
+ case UNKNOWN: // unknown are considered as add request to maintain
backward compatibility
+this.newSpecs.inc();
+this.jobCatalog.put(parsedMessage);
+break;
+ case DELETE:
+this.removedSpecs.inc();
+URI jobSpecUri = parsedMessage.getUri();
+this.jobCatalog.remove(jobSpecUri);
+// Delete the job state if it is a delete spec request
+deleteStateStore(jobSpecUri);
+break;
+ case CANCEL:
+this.jobCatalog.cancel(parsedMessage);
+break;
+ default:
Review comment:
Shall we throw exceptions here instead of just printing log ?
##
File path: gobblin-modules/gobblin-kafka-09/build.gradle
##
@@ -87,6 +87,9 @@ artifacts {
}
test {
+ // this module's test cases hang often when running in parallel.
+ // making every test class execute in a forked test process till tests are
fixed
+ forkEvery = 1
Review comment:
Q: Is this the only change targeting for the travis issue ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1198) use DatasetCleanerTask to clean job status
[ https://issues.apache.org/jira/browse/GOBBLIN-1198?focusedWorklogId=446773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446773 ] ASF GitHub Bot logged work on GOBBLIN-1198: --- Author: ASF GitHub Bot Created on: 16/Jun/20 17:21 Start Date: 16/Jun/20 17:21 Worklog Time Spent: 10m Work Description: codecov-commenter commented on pull request #3044: URL: https://github.com/apache/incubator-gobblin/pull/3044#issuecomment-644901118 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=h1) Report > Merging [#3044](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.01%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3044 +/- ## + Coverage 45.78% 45.79% +0.01% - Complexity 9328 9336 +8 Files 1956 1957 +1 Lines 7442374439 +16 Branches 8250 8252 +2 + Hits 3407334089 +16 - Misses3715737159 +2 + Partials 3193 3191 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/java/org/apache/gobblin/configuration/State.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9TdGF0ZS5qYXZh) | `28.26% <ø> (ø)` | `18.00 <0.00> (ø)` | | | [...blin/metastore/util/StateStoreCleanerRunnable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvU3RhdGVTdG9yZUNsZWFuZXJSdW5uYWJsZS5qYXZh) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | | | [...blin/service/monitoring/KafkaJobStatusMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUpvYlN0YXR1c01vbml0b3IuamF2YQ==) | `44.87% <0.00%> (-3.08%)` | `10.00 <0.00> (ø)` | | | [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `80.28% <0.00%> (-2.08%)` | `12.00% <0.00%> (ø%)` | | | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `62.79% <0.00%> (-1.50%)` | `6.00% <0.00%> (ø%)` | | | [...ache/gobblin/cluster/SingleFailInCreationTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlRmFpbEluQ3JlYXRpb25UYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `87.01% <0.00%> (+1.48%)` | `50.00% <0.00%> (+5.00%)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `49.51% <0.00%> (+1.94%)` | `12.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+5.55%)` | `16.00% <0.00%> (+2.00%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=continue). > **Legend** - [Click here to learn
[GitHub] [incubator-gobblin] codecov-commenter commented on pull request #3044: [GOBBLIN-1198] use DatasetCleanerTask to clean job status
codecov-commenter commented on pull request #3044: URL: https://github.com/apache/incubator-gobblin/pull/3044#issuecomment-644901118 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=h1) Report > Merging [#3044](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.01%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3044 +/- ## + Coverage 45.78% 45.79% +0.01% - Complexity 9328 9336 +8 Files 1956 1957 +1 Lines 7442374439 +16 Branches 8250 8252 +2 + Hits 3407334089 +16 - Misses3715737159 +2 + Partials 3193 3191 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/java/org/apache/gobblin/configuration/State.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9TdGF0ZS5qYXZh) | `28.26% <ø> (ø)` | `18.00 <0.00> (ø)` | | | [...blin/metastore/util/StateStoreCleanerRunnable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvU3RhdGVTdG9yZUNsZWFuZXJSdW5uYWJsZS5qYXZh) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | | | [...blin/service/monitoring/KafkaJobStatusMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUpvYlN0YXR1c01vbml0b3IuamF2YQ==) | `44.87% <0.00%> (-3.08%)` | `10.00 <0.00> (ø)` | | | [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `80.28% <0.00%> (-2.08%)` | `12.00% <0.00%> (ø%)` | | | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `62.79% <0.00%> (-1.50%)` | `6.00% <0.00%> (ø%)` | | | [...ache/gobblin/cluster/SingleFailInCreationTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlRmFpbEluQ3JlYXRpb25UYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `87.01% <0.00%> (+1.48%)` | `50.00% <0.00%> (+5.00%)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `49.51% <0.00%> (+1.94%)` | `12.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3044/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+5.55%)` | `16.00% <0.00%> (+2.00%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=footer). Last update [93ee795...6f8a0b5](https://codecov.io/gh/apache/incubator-gobblin/pull/3044?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[jira] [Work logged] (GOBBLIN-1127) Provide an option to make metric reporting instantiation failures fatal
[ https://issues.apache.org/jira/browse/GOBBLIN-1127?focusedWorklogId=446758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446758 ] ASF GitHub Bot logged work on GOBBLIN-1127: --- Author: ASF GitHub Bot Created on: 16/Jun/20 17:04 Start Date: 16/Jun/20 17:04 Worklog Time Spent: 10m Work Description: asfgit closed pull request #3035: URL: https://github.com/apache/incubator-gobblin/pull/3035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446758) Time Spent: 3h 10m (was: 3h) > Provide an option to make metric reporting instantiation failures fatal > --- > > Key: GOBBLIN-1127 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1127 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > This option allows GobblinTaskRunner to "fail-fast" on metric reporting > instantiation failures. This is particularly sseful in scenarios where > pipeline monitoring depends on metrics and tracking events being emitted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #3035: [GOBBLIN-1127] Provide an option to make metric reporting instantiation failure fatal
asfgit closed pull request #3035: URL: https://github.com/apache/incubator-gobblin/pull/3035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #3045: convert seconds to ms because helix api take time in ms
arjun4084346 opened a new pull request #3045:
URL: https://github.com/apache/incubator-gobblin/pull/3045
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1143) Add a generic wrapper producer client to communicate with Kafka
[ https://issues.apache.org/jira/browse/GOBBLIN-1143?focusedWorklogId=446240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446240 ] ASF GitHub Bot logged work on GOBBLIN-1143: --- Author: ASF GitHub Bot Created on: 16/Jun/20 02:33 Start Date: 16/Jun/20 02:33 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #2980: URL: https://github.com/apache/incubator-gobblin/pull/2980#issuecomment-644493665 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=h1) Report > Merging [#2980](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9029a89b85ef373f78d603b14d6aaa75998f3356=desc) will **increase** coverage by `0.18%`. > The diff coverage is `40.90%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2980 +/- ## + Coverage 45.61% 45.80% +0.18% - Complexity 9205 9342 +137 Files 1943 1957 +14 Lines 7373074440 +710 Branches 8138 8253 +115 + Hits 3363534100 +465 - Misses3696337153 +190 - Partials 3132 3187 +55 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/metrics/kafka/HighLevelGobblinKafkaProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvSGlnaExldmVsR29iYmxpbkthZmthUHJvZHVjZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...apache/gobblin/kafka/writer/Kafka08DataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2Evd3JpdGVyL0thZmthMDhEYXRhV3JpdGVyLmphdmE=) | `69.44% <60.00%> (-2.44%)` | `8.00 <1.00> (ø)` | | | [...apache/gobblin/kafka/writer/Kafka09DataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2Evd3JpdGVyL0thZmthMDlEYXRhV3JpdGVyLmphdmE=) | `74.57% <60.00%> (-2.21%)` | `10.00 <0.00> (+1.00)` | :arrow_down: | | [...g/apache/gobblin/service/NoopRequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9Ob29wUmVxdWVzdGVyU2VydmljZS5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-1.00%)` | | | [...a/org/apache/gobblin/service/RequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9SZXF1ZXN0ZXJTZXJ2aWNlLmphdmE=) | `52.94% <0.00%> (-39.37%)` | `3.00% <0.00%> (-1.00%)` | | | [...apache/gobblin/runtime/api/SpecSerDeException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL1NwZWNTZXJEZUV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-33.34%)` | `0.00% <0.00%> (-1.00%)` | | | [...ache/gobblin/runtime/spec\_serde/JavaSpecSerDe.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zZXJkZS9KYXZhU3BlY1NlckRlLmphdmE=) | `42.85% <0.00%> (-28.58%)` | `3.00% <0.00%> (ø%)` | | | [...che/gobblin/compaction/mapreduce/orc/OrcUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL29yYy9PcmNVdGlscy5qYXZh) | `52.70% <0.00%> (-20.22%)` | `38.00% <0.00%> (+26.00%)` | :arrow_down: | |
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[
https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446174
]
ASF GitHub Bot logged work on GOBBLIN-1193:
---
Author: ASF GitHub Bot
Created on: 15/Jun/20 23:46
Start Date: 15/Jun/20 23:46
Worklog Time Spent: 10m
Work Description: sv2000 opened a new pull request #3041:
URL: https://github.com/apache/incubator-gobblin/pull/3041
…consumed by Kafka Extractor
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1193
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
When no records are consumed by Kafka Extractor during an epoch,
KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
latency.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Added unit test in KafkaExtractorStatsTrackerTest
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446174)
Time Spent: 3h 40m (was: 3.5h)
> Ensure that ingestion latency is 0 when no records are consumed by Kafka
> Extractor
> --
>
> Key: GOBBLIN-1193
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1193
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-kafka
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Shirshanka Das
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> When no records are consumed by Kafka Extractor during an epoch,
> KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
> latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446175 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 23:46 Start Date: 15/Jun/20 23:46 Worklog Time Spent: 10m Work Description: sv2000 closed pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446175) Time Spent: 3h 50m (was: 3h 40m) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-kafka >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Shirshanka Das >Priority: Major > Fix For: 0.15.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > When no records are consumed by Kafka Extractor during an epoch, > KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion > latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446117 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 20:56 Start Date: 15/Jun/20 20:56 Worklog Time Spent: 10m Work Description: sv2000 closed pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446117) Time Spent: 2h 40m (was: 2.5h) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-kafka >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Shirshanka Das >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > When no records are consumed by Kafka Extractor during an epoch, > KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion > latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[
https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446118
]
ASF GitHub Bot logged work on GOBBLIN-1193:
---
Author: ASF GitHub Bot
Created on: 15/Jun/20 20:56
Start Date: 15/Jun/20 20:56
Worklog Time Spent: 10m
Work Description: sv2000 opened a new pull request #3041:
URL: https://github.com/apache/incubator-gobblin/pull/3041
…consumed by Kafka Extractor
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1193
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
When no records are consumed by Kafka Extractor during an epoch,
KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
latency.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Added unit test in KafkaExtractorStatsTrackerTest
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446118)
Time Spent: 2h 50m (was: 2h 40m)
> Ensure that ingestion latency is 0 when no records are consumed by Kafka
> Extractor
> --
>
> Key: GOBBLIN-1193
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1193
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-kafka
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Shirshanka Das
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> When no records are consumed by Kafka Extractor during an epoch,
> KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
> latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
Arjun Singh Bora created GOBBLIN-1196: - Summary: add support to search flow configs using multiple configs and other params Key: GOBBLIN-1196 URL: https://issues.apache.org/jira/browse/GOBBLIN-1196 Project: Apache Gobblin Issue Type: Improvement Reporter: Arjun Singh Bora -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
[ https://issues.apache.org/jira/browse/GOBBLIN-1196?focusedWorklogId=446393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446393 ] ASF GitHub Bot logged work on GOBBLIN-1196: --- Author: ASF GitHub Bot Created on: 16/Jun/20 09:55 Start Date: 16/Jun/20 09:55 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3042: URL: https://github.com/apache/incubator-gobblin/pull/3042#issuecomment-644437989 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=h1) Report > Merging [#3042](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.00%`. > The diff coverage is `45.05%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3042+/- ## == Coverage 45.78% 45.79% - Complexity 9328 9366+38 == Files 1956 1958 +2 Lines 7442374538 +115 Branches 8250 8277+27 == + Hits 3407334131+58 - Misses3715737194+37 - Partials 3193 3213+20 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/service/FlowConfigResourceLocalHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnUmVzb3VyY2VMb2NhbEhhbmRsZXIuamF2YQ==) | `18.82% <0.00%> (+2.98%)` | `2.00 <0.00> (ø)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `43.50% <0.00%> (-6.83%)` | `13.00 <0.00> (ø)` | | | [...va/org/apache/gobblin/runtime/api/SpecCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL1NwZWNDYXRhbG9nLmphdmE=) | `62.71% <0.00%> (-1.09%)` | `0.00 <0.00> (ø)` | | | [...ache/gobblin/runtime/spec\_catalog/FlowCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19jYXRhbG9nL0Zsb3dDYXRhbG9nLmphdmE=) | `47.18% <0.00%> (-2.09%)` | `17.00 <0.00> (ø)` | | | [.../gobblin/runtime/spec\_catalog/TopologyCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19jYXRhbG9nL1RvcG9sb2d5Q2F0YWxvZy5qYXZh) | `55.78% <ø> (ø)` | `12.00 <0.00> (ø)` | | | [...org/apache/gobblin/service/FlowConfigV2Client.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJDbGllbnQuamF2YQ==) | `60.22% <15.38%> (-9.40%)` | `10.00 <1.00> (ø)` | | | [...estli/GobblinServiceFlowConfigResourceHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9yZXN0bGkvR29iYmxpblNlcnZpY2VGbG93Q29uZmlnUmVzb3VyY2VIYW5kbGVyLmphdmE=) | `44.77% <50.00%> (+0.16%)` | `11.00 <1.00> (+1.00)` | | | [...che/gobblin/runtime/api/InstrumentedSpecStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0luc3RydW1lbnRlZFNwZWNTdG9yZS5qYXZh) | `63.49% <57.14%> (-0.80%)` | `9.00 <1.00>
[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #3044: [GOBBLIN-1198] use DatasetCleanerTask to clean job status
arjun4084346 opened a new pull request #3044:
URL: https://github.com/apache/incubator-gobblin/pull/3044
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1198
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
use DatasetCleanerTask to clean job status, because the old way, using
StateStoreCleanerRunnable, is deprecated
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial changes
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1198) use DatasetCleanerTask to clean job status
[
https://issues.apache.org/jira/browse/GOBBLIN-1198?focusedWorklogId=446579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446579
]
ASF GitHub Bot logged work on GOBBLIN-1198:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 15:19
Start Date: 16/Jun/20 15:19
Worklog Time Spent: 10m
Work Description: arjun4084346 opened a new pull request #3044:
URL: https://github.com/apache/incubator-gobblin/pull/3044
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1198
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
use DatasetCleanerTask to clean job status, because the old way, using
StateStoreCleanerRunnable, is deprecated
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial changes
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446579)
Remaining Estimate: 0h
Time Spent: 10m
> use DatasetCleanerTask to clean job status
> --
>
> Key: GOBBLIN-1198
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1198
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> use DatasetCleanerTask to clean job status, because the old way, using
> StateStoreCleanerRunnable, is deprecated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (GOBBLIN-1198) use DatasetCleanerTask to clean job status
Arjun Singh Bora created GOBBLIN-1198: - Summary: use DatasetCleanerTask to clean job status Key: GOBBLIN-1198 URL: https://issues.apache.org/jira/browse/GOBBLIN-1198 Project: Apache Gobblin Issue Type: Improvement Reporter: Arjun Singh Bora use DatasetCleanerTask to clean job status, because the old way, using StateStoreCleanerRunnable, is deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1192) Container commit suicide if Helix task creation failed after retry
[ https://issues.apache.org/jira/browse/GOBBLIN-1192?focusedWorklogId=446165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446165 ] ASF GitHub Bot logged work on GOBBLIN-1192: --- Author: ASF GitHub Bot Created on: 15/Jun/20 23:09 Start Date: 15/Jun/20 23:09 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3040: URL: https://github.com/apache/incubator-gobblin/pull/3040#issuecomment-643525522 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3040?src=pr=h1) Report > Merging [#3040](https://codecov.io/gh/apache/incubator-gobblin/pull/3040?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **decrease** coverage by `36.50%`. > The diff coverage is `26.66%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3040?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #3040 +/- ## - Coverage 45.78% 9.27% -36.51% + Complexity 93281696 -7632 Files 19561957+1 Lines 74423 74431+8 Branches 82508250 - Hits 340736906-27167 - Misses37157 66850+29693 + Partials 3193 675 -2518 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3040?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/gobblin/cluster/InMemoryWuFailedSingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSW5NZW1vcnlXdUZhaWxlZFNpbmdsZVRhc2suamF2YQ==) | `0.00% <ø> (-100.00%)` | `0.00 <0.00> (-3.00)` | | | [...g/apache/gobblin/cluster/InMemoryWuSingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSW5NZW1vcnlXdVNpbmdsZVRhc2suamF2YQ==) | `0.00% <ø> (-100.00%)` | `0.00 <0.00> (-3.00)` | | | [...ache/gobblin/cluster/SingleFailInCreationTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlRmFpbEluQ3JlYXRpb25UYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `59.30% <14.28%> (-4.99%)` | `5.00 <0.00> (-1.00)` | | | [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `63.38% <60.00%> (-18.98%)` | `8.00 <1.00> (-4.00)` | | | [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3040/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)`
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[
https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446077
]
ASF GitHub Bot logged work on GOBBLIN-1193:
---
Author: ASF GitHub Bot
Created on: 15/Jun/20 20:21
Start Date: 15/Jun/20 20:21
Worklog Time Spent: 10m
Work Description: sv2000 opened a new pull request #3041:
URL: https://github.com/apache/incubator-gobblin/pull/3041
…consumed by Kafka Extractor
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1193
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
When no records are consumed by Kafka Extractor during an epoch,
KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
latency.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Added unit test in KafkaExtractorStatsTrackerTest
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446077)
Time Spent: 2h 20m (was: 2h 10m)
> Ensure that ingestion latency is 0 when no records are consumed by Kafka
> Extractor
> --
>
> Key: GOBBLIN-1193
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1193
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-kafka
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Shirshanka Das
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When no records are consumed by Kafka Extractor during an epoch,
> KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
> latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446164 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 23:08 Start Date: 15/Jun/20 23:08 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041#issuecomment-643650145 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=h1) Report > Merging [#3041](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3041 +/- ## + Coverage 45.78% 45.79% +0.01% - Complexity 9328 9334 +6 Files 1956 1956 Lines 7442374426 +3 Branches 8250 8252 +2 + Hits 3407334086 +13 + Misses3715737148 -9 + Partials 3193 3192 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `87.01% <100.00%> (+1.48%)` | `50.00 <0.00> (+5.00)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `43.68% <0.00%> (-3.89%)` | `10.00% <0.00%> (-2.00%)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.56% <0.00%> (+0.66%)` | `25.00% <0.00%> (+1.00%)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0.00%> (+2.22%)` | `13.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+5.55%)` | `16.00% <0.00%> (+2.00%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=footer). Last update [93ee795...f123125](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446164) Time Spent: 3.5h (was: 3h 20m) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 >
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446078 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 20:21 Start Date: 15/Jun/20 20:21 Worklog Time Spent: 10m Work Description: sv2000 closed pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446078) Time Spent: 2.5h (was: 2h 20m) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-kafka >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Shirshanka Das >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > When no records are consumed by Kafka Extractor during an epoch, > KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion > latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3042: [GOBBLIN-1196] search flow configs using flow properties and/or other parameters
arjun4084346 commented on a change in pull request #3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440910388
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/SpecSearchObject.java
##
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.api;
+
+/**
+ * This is an interface to package all the parameters that should be used to
search {@link Spec} in a {@link SpecStore}
+ */
+public interface SpecSearchObject {
Review comment:
This is a "marker interface" to package parameters for search queries.
So far, FlowSpecSearchObject implements this, which has some unique set of
params suitable for a FlowSpec. In future, it can be implemented to package
params suitable for TopologySpec s
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
[
https://issues.apache.org/jira/browse/GOBBLIN-1196?focusedWorklogId=446557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446557
]
ASF GitHub Bot logged work on GOBBLIN-1196:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 14:48
Start Date: 16/Jun/20 14:48
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440910388
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/SpecSearchObject.java
##
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.api;
+
+/**
+ * This is an interface to package all the parameters that should be used to
search {@link Spec} in a {@link SpecStore}
+ */
+public interface SpecSearchObject {
Review comment:
This is a "marker interface" to package parameters for search queries.
So far, FlowSpecSearchObject implements this, which has some unique set of
params suitable for a FlowSpec. In future, it can be implemented to package
params suitable for TopologySpec s
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446557)
Time Spent: 50m (was: 40m)
> add support to search flow configs using multiple configs and other params
> --
>
> Key: GOBBLIN-1196
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1196
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
[
https://issues.apache.org/jira/browse/GOBBLIN-1196?focusedWorklogId=446555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446555
]
ASF GitHub Bot logged work on GOBBLIN-1196:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 14:45
Start Date: 16/Jun/20 14:45
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440908142
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
##
@@ -269,23 +270,40 @@ public boolean exists(URI uri) {
}
@Override
- public Spec getSpec(URI uri) throws SpecNotFoundException {
+ public Spec getSpecs(URI uri) throws SpecNotFoundException {
try {
return specStore.getSpec(uri);
} catch (IOException e) {
throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + uri, e);
}
}
+ @Override
+ public Collection getSpecs(SpecSearchObject specSearchObject) {
+try {
+ return specStore.getSpecs(specSearchObject);
+} catch (IOException e) {
+ throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + specSearchObject, e);
+}
+ }
+
+ public Collection getAllSpecs() {
+try {
+ return specStore.getSpecs();
Review comment:
Yes, good consideration. Also need to pay attention if slave nodes also
have all the specs in their memory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446555)
Time Spent: 40m (was: 0.5h)
> add support to search flow configs using multiple configs and other params
> --
>
> Key: GOBBLIN-1196
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1196
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #3042: [GOBBLIN-1196] search flow configs using flow properties and/or other parameters
arjun4084346 commented on a change in pull request #3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440908142
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
##
@@ -269,23 +270,40 @@ public boolean exists(URI uri) {
}
@Override
- public Spec getSpec(URI uri) throws SpecNotFoundException {
+ public Spec getSpecs(URI uri) throws SpecNotFoundException {
try {
return specStore.getSpec(uri);
} catch (IOException e) {
throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + uri, e);
}
}
+ @Override
+ public Collection getSpecs(SpecSearchObject specSearchObject) {
+try {
+ return specStore.getSpecs(specSearchObject);
+} catch (IOException e) {
+ throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + specSearchObject, e);
+}
+ }
+
+ public Collection getAllSpecs() {
+try {
+ return specStore.getSpecs();
Review comment:
Yes, good consideration. Also need to pay attention if slave nodes also
have all the specs in their memory.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
[
https://issues.apache.org/jira/browse/GOBBLIN-1196?focusedWorklogId=446551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446551
]
ASF GitHub Bot logged work on GOBBLIN-1196:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 14:35
Start Date: 16/Jun/20 14:35
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440893024
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/FlowSpec.java
##
@@ -411,5 +419,43 @@ public static String getFlowGroup(URI flowUri) {
}
return uriTokens[EXPECTED_NUM_URI_PATH_TOKENS - 2];
}
+
+public static FlowConfig toFlowConfig(Spec spec) {
Review comment:
Add javadoc here.
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/SpecSearchObject.java
##
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.api;
+
+/**
+ * This is an interface to package all the parameters that should be used to
search {@link Spec} in a {@link SpecStore}
+ */
+public interface SpecSearchObject {
Review comment:
Why define an empty interface?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
##
@@ -269,23 +270,40 @@ public boolean exists(URI uri) {
}
@Override
- public Spec getSpec(URI uri) throws SpecNotFoundException {
+ public Spec getSpecs(URI uri) throws SpecNotFoundException {
try {
return specStore.getSpec(uri);
} catch (IOException e) {
throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + uri, e);
}
}
+ @Override
+ public Collection getSpecs(SpecSearchObject specSearchObject) {
+try {
+ return specStore.getSpecs(specSearchObject);
+} catch (IOException e) {
+ throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + specSearchObject, e);
+}
+ }
+
+ public Collection getAllSpecs() {
+try {
+ return specStore.getSpecs();
Review comment:
An option to consider: GobblinServiceScheduler already caches all the
specs in memory. So we should be able to avoid going to the SpecStore to
retrieve all the specs each time getAllSpecs() is invoked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446551)
Time Spent: 0.5h (was: 20m)
> add support to search flow configs using multiple configs and other params
> --
>
> Key: GOBBLIN-1196
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1196
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #3042: [GOBBLIN-1196] search flow configs using flow properties and/or other parameters
sv2000 commented on a change in pull request #3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440893024
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/FlowSpec.java
##
@@ -411,5 +419,43 @@ public static String getFlowGroup(URI flowUri) {
}
return uriTokens[EXPECTED_NUM_URI_PATH_TOKENS - 2];
}
+
+public static FlowConfig toFlowConfig(Spec spec) {
Review comment:
Add javadoc here.
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/SpecSearchObject.java
##
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.api;
+
+/**
+ * This is an interface to package all the parameters that should be used to
search {@link Spec} in a {@link SpecStore}
+ */
+public interface SpecSearchObject {
Review comment:
Why define an empty interface?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
##
@@ -269,23 +270,40 @@ public boolean exists(URI uri) {
}
@Override
- public Spec getSpec(URI uri) throws SpecNotFoundException {
+ public Spec getSpecs(URI uri) throws SpecNotFoundException {
try {
return specStore.getSpec(uri);
} catch (IOException e) {
throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + uri, e);
}
}
+ @Override
+ public Collection getSpecs(SpecSearchObject specSearchObject) {
+try {
+ return specStore.getSpecs(specSearchObject);
+} catch (IOException e) {
+ throw new RuntimeException("Cannot retrieve Spec from Spec store for
URI: " + specSearchObject, e);
+}
+ }
+
+ public Collection getAllSpecs() {
+try {
+ return specStore.getSpecs();
Review comment:
An option to consider: GobblinServiceScheduler already caches all the
specs in memory. So we should be able to avoid going to the SpecStore to
retrieve all the specs each time getAllSpecs() is invoked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter edited a comment on pull request #3042: [GOBBLIN-1196] search flow configs using flow properties and/or other parameters
codecov-commenter edited a comment on pull request #3042: URL: https://github.com/apache/incubator-gobblin/pull/3042#issuecomment-644437989 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=h1) Report > Merging [#3042](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.00%`. > The diff coverage is `45.05%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3042+/- ## == Coverage 45.78% 45.79% - Complexity 9328 9366+38 == Files 1956 1958 +2 Lines 7442374538 +115 Branches 8250 8277+27 == + Hits 3407334131+58 - Misses3715737194+37 - Partials 3193 3213+20 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3042?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/service/FlowConfigResourceLocalHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnUmVzb3VyY2VMb2NhbEhhbmRsZXIuamF2YQ==) | `18.82% <0.00%> (+2.98%)` | `2.00 <0.00> (ø)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `43.50% <0.00%> (-6.83%)` | `13.00 <0.00> (ø)` | | | [...va/org/apache/gobblin/runtime/api/SpecCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL1NwZWNDYXRhbG9nLmphdmE=) | `62.71% <0.00%> (-1.09%)` | `0.00 <0.00> (ø)` | | | [...ache/gobblin/runtime/spec\_catalog/FlowCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19jYXRhbG9nL0Zsb3dDYXRhbG9nLmphdmE=) | `47.18% <0.00%> (-2.09%)` | `17.00 <0.00> (ø)` | | | [.../gobblin/runtime/spec\_catalog/TopologyCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19jYXRhbG9nL1RvcG9sb2d5Q2F0YWxvZy5qYXZh) | `55.78% <ø> (ø)` | `12.00 <0.00> (ø)` | | | [...org/apache/gobblin/service/FlowConfigV2Client.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJDbGllbnQuamF2YQ==) | `60.22% <15.38%> (-9.40%)` | `10.00 <1.00> (ø)` | | | [...estli/GobblinServiceFlowConfigResourceHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9yZXN0bGkvR29iYmxpblNlcnZpY2VGbG93Q29uZmlnUmVzb3VyY2VIYW5kbGVyLmphdmE=) | `44.77% <50.00%> (+0.16%)` | `11.00 <1.00> (+1.00)` | | | [...che/gobblin/runtime/api/InstrumentedSpecStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0luc3RydW1lbnRlZFNwZWNTdG9yZS5qYXZh) | `63.49% <57.14%> (-0.80%)` | `9.00 <1.00> (+1.00)` | :arrow_down: | | [...che/gobblin/runtime/spec\_store/MysqlSpecStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3042/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9NeXNxbFNwZWNTdG9yZS5qYXZh) | `65.38% <58.58%> (-8.98%)` | `28.00 <16.00> (+14.00)` | :arrow_down: | | ... and [16
[jira] [Work logged] (GOBBLIN-1127) Provide an option to make metric reporting instantiation failures fatal
[ https://issues.apache.org/jira/browse/GOBBLIN-1127?focusedWorklogId=446213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446213 ] ASF GitHub Bot logged work on GOBBLIN-1127: --- Author: ASF GitHub Bot Created on: 16/Jun/20 01:13 Start Date: 16/Jun/20 01:13 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3035: URL: https://github.com/apache/incubator-gobblin/pull/3035#issuecomment-642214083 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3035?src=pr=h1) Report > Merging [#3035](https://codecov.io/gh/apache/incubator-gobblin/pull/3035?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/31f2ef1478c3b5d4887cfc772f0e85057cde1c56=desc) will **increase** coverage by `0.13%`. > The diff coverage is `25.78%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3035?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3035 +/- ## + Coverage 45.66% 45.80% +0.13% - Complexity 9296 9347 +51 Files 1956 1961 +5 Lines 7439774547 +150 Branches 8247 8262 +15 + Hits 3397734145 +168 + Misses3724237203 -39 - Partials 3178 3199 +21 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3035?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | | | [...ache/gobblin/compaction/mapreduce/MRCompactor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL01SQ29tcGFjdG9yLmphdmE=) | `7.04% <0.00%> (-0.10%)` | `9.00 <0.00> (ø)` | | | [...pache/gobblin/metrics/MetricReporterException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9NZXRyaWNSZXBvcnRlckV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...apache/gobblin/metrics/MultiReporterException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9NdWx0aVJlcG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...a/org/apache/gobblin/metrics/ReporterSinkType.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9SZXBvcnRlclNpbmtUeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../java/org/apache/gobblin/metrics/ReporterType.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9SZXBvcnRlclR5cGUuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...bblin/metrics/reporter/util/MetricReportUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci91dGlsL01ldHJpY1JlcG9ydFV0aWxzLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...org/apache/gobblin/azkaban/AzkabanJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Kb2JMYXVuY2hlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...bblin/metrics/kafka/KafkaEventReporterFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3035/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | |
[jira] [Created] (GOBBLIN-1197) Fix race contention among different tests for port allocation
Lei Sun created GOBBLIN-1197: Summary: Fix race contention among different tests for port allocation Key: GOBBLIN-1197 URL: https://issues.apache.org/jira/browse/GOBBLIN-1197 Project: Apache Gobblin Issue Type: Improvement Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1197) Fix race contention among different tests for port allocation
[
https://issues.apache.org/jira/browse/GOBBLIN-1197?focusedWorklogId=446298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446298
]
ASF GitHub Bot logged work on GOBBLIN-1197:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 05:31
Start Date: 16/Jun/20 05:31
Worklog Time Spent: 10m
Work Description: autumnust opened a new pull request #3043:
URL: https://github.com/apache/incubator-gobblin/pull/3043
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1197
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446298)
Remaining Estimate: 0h
Time Spent: 10m
> Fix race contention among different tests for port allocation
> -
>
> Key: GOBBLIN-1197
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1197
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1143) Add a generic wrapper producer client to communicate with Kafka
[ https://issues.apache.org/jira/browse/GOBBLIN-1143?focusedWorklogId=446239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446239 ] ASF GitHub Bot logged work on GOBBLIN-1143: --- Author: ASF GitHub Bot Created on: 16/Jun/20 02:31 Start Date: 16/Jun/20 02:31 Worklog Time Spent: 10m Work Description: codecov-commenter commented on pull request #2980: URL: https://github.com/apache/incubator-gobblin/pull/2980#issuecomment-644493665 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=h1) Report > Merging [#2980](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9029a89b85ef373f78d603b14d6aaa75998f3356=desc) will **increase** coverage by `0.18%`. > The diff coverage is `40.90%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2980 +/- ## + Coverage 45.61% 45.80% +0.18% - Complexity 9205 9342 +137 Files 1943 1957 +14 Lines 7373074440 +710 Branches 8138 8253 +115 + Hits 3363534100 +465 - Misses3696337153 +190 - Partials 3132 3187 +55 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2980?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/metrics/kafka/HighLevelGobblinKafkaProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvSGlnaExldmVsR29iYmxpbkthZmthUHJvZHVjZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...apache/gobblin/kafka/writer/Kafka08DataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2Evd3JpdGVyL0thZmthMDhEYXRhV3JpdGVyLmphdmE=) | `69.44% <60.00%> (-2.44%)` | `8.00 <1.00> (ø)` | | | [...apache/gobblin/kafka/writer/Kafka09DataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2Evd3JpdGVyL0thZmthMDlEYXRhV3JpdGVyLmphdmE=) | `74.57% <60.00%> (-2.21%)` | `10.00 <0.00> (+1.00)` | :arrow_down: | | [...g/apache/gobblin/service/NoopRequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9Ob29wUmVxdWVzdGVyU2VydmljZS5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-1.00%)` | | | [...a/org/apache/gobblin/service/RequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9SZXF1ZXN0ZXJTZXJ2aWNlLmphdmE=) | `52.94% <0.00%> (-39.37%)` | `3.00% <0.00%> (-1.00%)` | | | [...apache/gobblin/runtime/api/SpecSerDeException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL1NwZWNTZXJEZUV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (-33.34%)` | `0.00% <0.00%> (-1.00%)` | | | [...ache/gobblin/runtime/spec\_serde/JavaSpecSerDe.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zZXJkZS9KYXZhU3BlY1NlckRlLmphdmE=) | `42.85% <0.00%> (-28.58%)` | `3.00% <0.00%> (ø%)` | | | [...che/gobblin/compaction/mapreduce/orc/OrcUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2980/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL29yYy9PcmNVdGlscy5qYXZh) | `52.70% <0.00%> (-20.22%)` | `38.00% <0.00%> (+26.00%)` | :arrow_down: | |
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446160 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 22:56 Start Date: 15/Jun/20 22:56 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041#issuecomment-643650145 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=h1) Report > Merging [#3041](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **decrease** coverage by `36.50%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #3041 +/- ## - Coverage 45.78% 9.27% -36.51% + Complexity 93281696 -7632 Files 19561956 Lines 74423 74426+3 Branches 82508252+2 - Hits 340736904-27169 - Misses37157 66848+29691 + Partials 3193 674 -2519 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `0.00% <0.00%> (-85.53%)` | `0.00 <0.00> (-45.00)` | | | [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [.../org/apache/gobblin/yarn/HelixMessageSubTypes.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vSGVsaXhNZXNzYWdlU3ViVHlwZXMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...va/org/apache/gobblin/cluster/SingleHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlSGVsaXhUYXNrLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | |
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446200 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 16/Jun/20 00:39 Start Date: 16/Jun/20 00:39 Worklog Time Spent: 10m Work Description: asfgit closed pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446200) Time Spent: 4h 10m (was: 4h) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-kafka >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Shirshanka Das >Priority: Major > Fix For: 0.15.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > When no records are consumed by Kafka Extractor during an epoch, > KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion > latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1196) add support to search flow configs using multiple configs and other params
[
https://issues.apache.org/jira/browse/GOBBLIN-1196?focusedWorklogId=446348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446348
]
ASF GitHub Bot logged work on GOBBLIN-1196:
---
Author: ASF GitHub Bot
Created on: 16/Jun/20 07:41
Start Date: 16/Jun/20 07:41
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440511681
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-client/src/main/java/org/apache/gobblin/service/FlowConfigV2Client.java
##
@@ -180,17 +184,48 @@ public void partialUpdateFlowConfig(FlowId flowId,
PatchRequest flow
*/
public FlowConfig getFlowConfig(FlowId flowId)
throws RemoteInvocationException {
-LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " +
-flowId.getFlowName());
+LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " + flowId.getFlowName());
GetRequest getRequest = _flowconfigsV2RequestBuilders.get()
.id(new ComplexResourceKey<>(flowId, new FlowStatusId())).build();
-Response response =
-_restClient.get().sendRequest(getRequest).getResponse();
+Response response =
_restClient.get().sendRequest(getRequest).getResponse();
return response.getEntity();
}
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getAllFlowConfigs() throws
RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+GetAllRequest getRequest =
_flowconfigsV2RequestBuilders.getAll().build();
+Response> response =
_restClient.get().sendRequest(getRequest).getResponse();
+return response.getEntity().getElements();
+ }
+
+ /**
+ * Get all {@link FlowConfig}s
Review comment:
Can you make this comment more descriptive, it's get all flow configs
but with a filter right?
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-client/src/main/java/org/apache/gobblin/service/FlowConfigV2Client.java
##
@@ -180,17 +184,48 @@ public void partialUpdateFlowConfig(FlowId flowId,
PatchRequest flow
*/
public FlowConfig getFlowConfig(FlowId flowId)
throws RemoteInvocationException {
-LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " +
-flowId.getFlowName());
+LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " + flowId.getFlowName());
GetRequest getRequest = _flowconfigsV2RequestBuilders.get()
.id(new ComplexResourceKey<>(flowId, new FlowStatusId())).build();
-Response response =
-_restClient.get().sendRequest(getRequest).getResponse();
+Response response =
_restClient.get().sendRequest(getRequest).getResponse();
return response.getEntity();
}
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getAllFlowConfigs() throws
RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+GetAllRequest getRequest =
_flowconfigsV2RequestBuilders.getAll().build();
+Response> response =
_restClient.get().sendRequest(getRequest).getResponse();
+return response.getEntity().getElements();
+ }
+
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getFlowConfigs(String flowGroup, String
flowName, String templateUri, String userToProxy,
+ String sourceIdentifier, String destinationIdentifier, String schedule,
Boolean isRunImmediately, String owningGroup,
+ String propertyFilter) throws RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+FindRequest getRequest =
_flowconfigsV2RequestBuilders.findByFilterFlows()
Review comment:
Is `propertyFilter` meant to be passed here as well?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##
@@ -267,7 +276,128 @@ public Spec getSpec(URI specUri, String version) throws
IOException, SpecNotFoun
return Optional.of(this.specStoreURI);
}
- protected void setPreparedStatement(PreparedStatement statement, Spec spec,
String tagValue) throws SQLException {
+ static String createGetPreparedStatement(FlowSpecSearchObject
flowSpecSearchObject, String tableName)
+ throws IOException {
+String baseStatement = String.format(GET_STATEMENT, tableName);
+List conditions = new ArrayList<>();
+
+if (flowSpecSearchObject.getFlowSpecUri() != null) {
+ conditions.add("spec_uri = ?");
+}
+
+if
[GitHub] [incubator-gobblin] jack-moseley commented on a change in pull request #3042: [GOBBLIN-1196] search flow configs using flow properties and/or other parameters
jack-moseley commented on a change in pull request #3042:
URL: https://github.com/apache/incubator-gobblin/pull/3042#discussion_r440511681
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-client/src/main/java/org/apache/gobblin/service/FlowConfigV2Client.java
##
@@ -180,17 +184,48 @@ public void partialUpdateFlowConfig(FlowId flowId,
PatchRequest flow
*/
public FlowConfig getFlowConfig(FlowId flowId)
throws RemoteInvocationException {
-LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " +
-flowId.getFlowName());
+LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " + flowId.getFlowName());
GetRequest getRequest = _flowconfigsV2RequestBuilders.get()
.id(new ComplexResourceKey<>(flowId, new FlowStatusId())).build();
-Response response =
-_restClient.get().sendRequest(getRequest).getResponse();
+Response response =
_restClient.get().sendRequest(getRequest).getResponse();
return response.getEntity();
}
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getAllFlowConfigs() throws
RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+GetAllRequest getRequest =
_flowconfigsV2RequestBuilders.getAll().build();
+Response> response =
_restClient.get().sendRequest(getRequest).getResponse();
+return response.getEntity().getElements();
+ }
+
+ /**
+ * Get all {@link FlowConfig}s
Review comment:
Can you make this comment more descriptive, it's get all flow configs
but with a filter right?
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-client/src/main/java/org/apache/gobblin/service/FlowConfigV2Client.java
##
@@ -180,17 +184,48 @@ public void partialUpdateFlowConfig(FlowId flowId,
PatchRequest flow
*/
public FlowConfig getFlowConfig(FlowId flowId)
throws RemoteInvocationException {
-LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " +
-flowId.getFlowName());
+LOG.debug("getFlowConfig with groupName " + flowId.getFlowGroup() + "
flowName " + flowId.getFlowName());
GetRequest getRequest = _flowconfigsV2RequestBuilders.get()
.id(new ComplexResourceKey<>(flowId, new FlowStatusId())).build();
-Response response =
-_restClient.get().sendRequest(getRequest).getResponse();
+Response response =
_restClient.get().sendRequest(getRequest).getResponse();
return response.getEntity();
}
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getAllFlowConfigs() throws
RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+GetAllRequest getRequest =
_flowconfigsV2RequestBuilders.getAll().build();
+Response> response =
_restClient.get().sendRequest(getRequest).getResponse();
+return response.getEntity().getElements();
+ }
+
+ /**
+ * Get all {@link FlowConfig}s
+ * @return all {@link FlowConfig}s
+ * @throws RemoteInvocationException
+ */
+ public Collection getFlowConfigs(String flowGroup, String
flowName, String templateUri, String userToProxy,
+ String sourceIdentifier, String destinationIdentifier, String schedule,
Boolean isRunImmediately, String owningGroup,
+ String propertyFilter) throws RemoteInvocationException {
+LOG.debug("getAllFlowConfigs called");
+
+FindRequest getRequest =
_flowconfigsV2RequestBuilders.findByFilterFlows()
Review comment:
Is `propertyFilter` meant to be passed here as well?
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##
@@ -267,7 +276,128 @@ public Spec getSpec(URI specUri, String version) throws
IOException, SpecNotFoun
return Optional.of(this.specStoreURI);
}
- protected void setPreparedStatement(PreparedStatement statement, Spec spec,
String tagValue) throws SQLException {
+ static String createGetPreparedStatement(FlowSpecSearchObject
flowSpecSearchObject, String tableName)
+ throws IOException {
+String baseStatement = String.format(GET_STATEMENT, tableName);
+List conditions = new ArrayList<>();
+
+if (flowSpecSearchObject.getFlowSpecUri() != null) {
+ conditions.add("spec_uri = ?");
+}
+
+if (flowSpecSearchObject.getFlowGroup() != null) {
+ conditions.add("flow_group = ?");
+}
+
+if (flowSpecSearchObject.getFlowName() != null) {
+ conditions.add("flow_name = ?");
+}
+
+if (flowSpecSearchObject.getTemplateURI() != null) {
+ conditions.add("template_uri = ?");
+}
+
+if (flowSpecSearchObject.getUserToProxy() != null) {
+ conditions.add("user_to_proxy = ?");
+}
+
+if
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[
https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446156
]
ASF GitHub Bot logged work on GOBBLIN-1193:
---
Author: ASF GitHub Bot
Created on: 15/Jun/20 22:37
Start Date: 15/Jun/20 22:37
Worklog Time Spent: 10m
Work Description: sv2000 opened a new pull request #3041:
URL: https://github.com/apache/incubator-gobblin/pull/3041
…consumed by Kafka Extractor
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1193
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
When no records are consumed by Kafka Extractor during an epoch,
KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
latency.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Added unit test in KafkaExtractorStatsTrackerTest
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 446156)
Time Spent: 3h (was: 2h 50m)
> Ensure that ingestion latency is 0 when no records are consumed by Kafka
> Extractor
> --
>
> Key: GOBBLIN-1193
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1193
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-kafka
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Shirshanka Das
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> When no records are consumed by Kafka Extractor during an epoch,
> KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion
> latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446157 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 15/Jun/20 22:37 Start Date: 15/Jun/20 22:37 Worklog Time Spent: 10m Work Description: sv2000 closed pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446157) Time Spent: 3h 10m (was: 3h) > Ensure that ingestion latency is 0 when no records are consumed by Kafka > Extractor > -- > > Key: GOBBLIN-1193 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1193 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-kafka >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Shirshanka Das >Priority: Major > Fix For: 0.15.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > When no records are consumed by Kafka Extractor during an epoch, > KafkaExtractorStatsTracker incorrectly returns a non-zero value for ingestion > latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor
[ https://issues.apache.org/jira/browse/GOBBLIN-1193?focusedWorklogId=446199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446199 ] ASF GitHub Bot logged work on GOBBLIN-1193: --- Author: ASF GitHub Bot Created on: 16/Jun/20 00:28 Start Date: 16/Jun/20 00:28 Worklog Time Spent: 10m Work Description: codecov-commenter edited a comment on pull request #3041: URL: https://github.com/apache/incubator-gobblin/pull/3041#issuecomment-643650145 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=h1) Report > Merging [#3041](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/93ee79546759d73310fb6f949fcbfc57e232ccd8=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3041 +/- ## + Coverage 45.78% 45.79% +0.01% - Complexity 9328 9335 +7 Files 1956 1956 Lines 7442374426 +3 Branches 8250 8252 +2 + Hits 3407334087 +14 + Misses3715737148 -9 + Partials 3193 3191 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `87.01% <100.00%> (+1.48%)` | `50.00 <0.00> (+5.00)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `43.68% <0.00%> (-3.89%)` | `10.00% <0.00%> (-2.00%)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.03% <0.00%> (+0.33%)` | `33.00% <0.00%> (ø%)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.56% <0.00%> (+0.66%)` | `25.00% <0.00%> (+1.00%)` | | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0.00%> (+2.22%)` | `13.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3041/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+5.55%)` | `16.00% <0.00%> (+2.00%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=footer). Last update [93ee795...f123125](https://codecov.io/gh/apache/incubator-gobblin/pull/3041?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446199) Time Spent: 4h (was:
[jira] [Work logged] (GOBBLIN-1192) Container commit suicide if Helix task creation failed after retry
[ https://issues.apache.org/jira/browse/GOBBLIN-1192?focusedWorklogId=446282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446282 ] ASF GitHub Bot logged work on GOBBLIN-1192: --- Author: ASF GitHub Bot Created on: 16/Jun/20 04:48 Start Date: 16/Jun/20 04:48 Worklog Time Spent: 10m Work Description: asfgit closed pull request #3040: URL: https://github.com/apache/incubator-gobblin/pull/3040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 446282) Time Spent: 1h (was: 50m) > Container commit suicide if Helix task creation failed after retry > -- > > Key: GOBBLIN-1192 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1192 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1195) Close the writer when a fork is done
Hung Tran created GOBBLIN-1195: -- Summary: Close the writer when a fork is done Key: GOBBLIN-1195 URL: https://issues.apache.org/jira/browse/GOBBLIN-1195 Project: Apache Gobblin Issue Type: Improvement Reporter: Hung Tran -- This message was sent by Atlassian Jira (v8.3.4#803005)
