[GitHub] [incubator-gobblin] jhsenjaliya commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type
jhsenjaliya commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type URL: https://github.com/apache/incubator-gobblin/pull/2691#issuecomment-528501611 @arjun4084346 , i made whatever changes i had to make to facilitate the backward compatibility, please take a look. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-830) update config key to define job launcher type
[ https://issues.apache.org/jira/browse/GOBBLIN-830?focusedWorklogId=307377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307377 ] ASF GitHub Bot logged work on GOBBLIN-830: -- Author: ASF GitHub Bot Created on: 05/Sep/19 17:55 Start Date: 05/Sep/19 17:55 Worklog Time Spent: 10m Work Description: jhsenjaliya commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type URL: https://github.com/apache/incubator-gobblin/pull/2691#issuecomment-528501611 @arjun4084346 , i made whatever changes i had to make to facilitate the backward compatibility, please take a look. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307377) Time Spent: 1h 10m (was: 1h) > update config key to define job launcher type > - > > Key: GOBBLIN-830 > URL: https://issues.apache.org/jira/browse/GOBBLIN-830 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jay Sen >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > config key JOB_LAUNCHER_TYPE_KEY = "launcher.type" should be set to > "job.launcher.type" -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-830) update config key to define job launcher type
[ https://issues.apache.org/jira/browse/GOBBLIN-830?focusedWorklogId=307373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307373 ] ASF GitHub Bot logged work on GOBBLIN-830: -- Author: ASF GitHub Bot Created on: 05/Sep/19 17:50 Start Date: 05/Sep/19 17:50 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type URL: https://github.com/apache/incubator-gobblin/pull/2691#issuecomment-528496312 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=h1) Report > Merging [#2691](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `46.15%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2691 +/- ## + Coverage 45.01% 45.01% +<.01% - Complexity 8740 8742 +2 Files 1884 1885 +1 Lines 7026670276 +10 Branches 7709 7712 +3 + Hits 3162831635 +7 - Misses3570935714 +5 + Partials 2929 2927 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/runtime/job\_exec/JobLauncherExecutionDriver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2V4ZWMvSm9iTGF1bmNoZXJFeGVjdXRpb25Ecml2ZXIuamF2YQ==) | `66.66% <ø> (ø)` | `14 <0> (ø)` | :arrow_down: | | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...org/apache/gobblin/azkaban/AzkabanJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Kb2JMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...java/org/apache/gobblin/util/JobLauncherUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSm9iTGF1bmNoZXJVdGlscy5qYXZh) | `20.9% <0%> (-1.87%)` | `10 <0> (ø)` | | | [...ache/gobblin/runtime/embedded/EmbeddedGobblin.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvZW1iZWRkZWQvRW1iZWRkZWRHb2JibGluLmphdmE=) | `56.63% <0%> (ø)` | `16 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.57% <100%> (ø)` | `58 <0> (ø)` | :arrow_down: | | [...va/org/apache/gobblin/runtime/JobLauncherType.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iTGF1bmNoZXJUeXBlLmphdmE=) | `100% <100%> (ø)` | `1 <1> (?)` | | | [...org/apache/gobblin/runtime/JobLauncherFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iTGF1bmNoZXJGYWN0b3J5LmphdmE=) | `52.38% <87.5%> (-5.96%)` | `7 <2> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `66.19% <0%> (+0.46%)` | `29% <0%> (ø)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=continue). > **Legend** - [Click
[GitHub] [incubator-gobblin] codecov-io commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type
codecov-io commented on issue #2691: [GOBBLIN-830] launcher.type to job.launcher.type URL: https://github.com/apache/incubator-gobblin/pull/2691#issuecomment-528496312 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=h1) Report > Merging [#2691](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `46.15%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2691 +/- ## + Coverage 45.01% 45.01% +<.01% - Complexity 8740 8742 +2 Files 1884 1885 +1 Lines 7026670276 +10 Branches 7709 7712 +3 + Hits 3162831635 +7 - Misses3570935714 +5 + Partials 2929 2927 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...n/runtime/job\_exec/JobLauncherExecutionDriver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2V4ZWMvSm9iTGF1bmNoZXJFeGVjdXRpb25Ecml2ZXIuamF2YQ==) | `66.66% <ø> (ø)` | `14 <0> (ø)` | :arrow_down: | | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...org/apache/gobblin/azkaban/AzkabanJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Kb2JMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...java/org/apache/gobblin/util/JobLauncherUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSm9iTGF1bmNoZXJVdGlscy5qYXZh) | `20.9% <0%> (-1.87%)` | `10 <0> (ø)` | | | [...ache/gobblin/runtime/embedded/EmbeddedGobblin.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvZW1iZWRkZWQvRW1iZWRkZWRHb2JibGluLmphdmE=) | `56.63% <0%> (ø)` | `16 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.57% <100%> (ø)` | `58 <0> (ø)` | :arrow_down: | | [...va/org/apache/gobblin/runtime/JobLauncherType.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iTGF1bmNoZXJUeXBlLmphdmE=) | `100% <100%> (ø)` | `1 <1> (?)` | | | [...org/apache/gobblin/runtime/JobLauncherFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iTGF1bmNoZXJGYWN0b3J5LmphdmE=) | `52.38% <87.5%> (-5.96%)` | `7 <2> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `66.19% <0%> (+0.46%)` | `29% <0%> (ø)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2691/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=footer). Last update [3523dbc...49818ef](https://codecov.io/gh/apache/incubator-gobblin/pull/2691?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[jira] [Created] (GOBBLIN-867) use logger from class instead of passing it around
Jay Sen created GOBBLIN-867: --- Summary: use logger from class instead of passing it around Key: GOBBLIN-867 URL: https://issues.apache.org/jira/browse/GOBBLIN-867 Project: Apache Gobblin Issue Type: Improvement Reporter: Jay Sen in some classes, mainly static methods, expects the logger, so user of the methods has to pass the logger, these static methods are utils anyway, should just have their own logger. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
[
https://issues.apache.org/jira/browse/GOBBLIN-868?focusedWorklogId=307502=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307502
]
ASF GitHub Bot logged work on GOBBLIN-868:
--
Author: ASF GitHub Bot
Created on: 05/Sep/19 22:47
Start Date: 05/Sep/19 22:47
Worklog Time Spent: 10m
Work Description: jack-moseley commented on pull request #2723:
[GOBBLIN-868] Check flow status instead of job status to determine if flow is
running
URL: https://github.com/apache/incubator-gobblin/pull/2723
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-868
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
After updating to use flow level events for job status, this code should be
using the flow level status instead of using jobs.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307502)
Remaining Estimate: 0h
Time Spent: 10m
> Job statuses are still used for determining if a flow is running
>
>
> Key: GOBBLIN-868
> URL: https://issues.apache.org/jira/browse/GOBBLIN-868
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running
jack-moseley opened a new pull request #2723: [GOBBLIN-868] Check flow status
instead of job status to determine if flow is running
URL: https://github.com/apache/incubator-gobblin/pull/2723
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-868
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
After updating to use flow level events for job status, this code should be
using the flow level status instead of using jobs.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running
jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528620952 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
[ https://issues.apache.org/jira/browse/GOBBLIN-868?focusedWorklogId=307503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307503 ] ASF GitHub Bot logged work on GOBBLIN-868: -- Author: ASF GitHub Bot Created on: 05/Sep/19 22:47 Start Date: 05/Sep/19 22:47 Worklog Time Spent: 10m Work Description: jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528620952 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307503) Time Spent: 20m (was: 10m) > Job statuses are still used for determining if a flow is running > > > Key: GOBBLIN-868 > URL: https://issues.apache.org/jira/browse/GOBBLIN-868 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
Jack Moseley created GOBBLIN-868: Summary: Job statuses are still used for determining if a flow is running Key: GOBBLIN-868 URL: https://issues.apache.org/jira/browse/GOBBLIN-868 Project: Apache Gobblin Issue Type: Bug Reporter: Jack Moseley -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
[ https://issues.apache.org/jira/browse/GOBBLIN-868?focusedWorklogId=307527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307527 ] ASF GitHub Bot logged work on GOBBLIN-868: -- Author: ASF GitHub Bot Created on: 05/Sep/19 23:24 Start Date: 05/Sep/19 23:24 Worklog Time Spent: 10m Work Description: jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528644372 @sv2000 please merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307527) Time Spent: 40m (was: 0.5h) > Job statuses are still used for determining if a flow is running > > > Key: GOBBLIN-868 > URL: https://issues.apache.org/jira/browse/GOBBLIN-868 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running
jack-moseley commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528644372 @sv2000 please merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
[ https://issues.apache.org/jira/browse/GOBBLIN-868?focusedWorklogId=307526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307526 ] ASF GitHub Bot logged work on GOBBLIN-868: -- Author: ASF GitHub Bot Created on: 05/Sep/19 23:23 Start Date: 05/Sep/19 23:23 Worklog Time Spent: 10m Work Description: arjun4084346 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528644066 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307526) Time Spent: 0.5h (was: 20m) > Job statuses are still used for determining if a flow is running > > > Key: GOBBLIN-868 > URL: https://issues.apache.org/jira/browse/GOBBLIN-868 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-869) Update logging to include entire stack trace when adding a datanode fails
[
https://issues.apache.org/jira/browse/GOBBLIN-869?focusedWorklogId=307540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307540
]
ASF GitHub Bot logged work on GOBBLIN-869:
--
Author: ASF GitHub Bot
Created on: 05/Sep/19 23:51
Start Date: 05/Sep/19 23:51
Worklog Time Spent: 10m
Work Description: jack-moseley commented on pull request #2724:
[GOBBLIN-869] Update logging to include entire stack trace when adding a
datanode f…
URL: https://github.com/apache/incubator-gobblin/pull/2724
…ails
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-869
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307540)
Remaining Estimate: 0h
Time Spent: 10m
> Update logging to include entire stack trace when adding a datanode fails
> -
>
> Key: GOBBLIN-869
> URL: https://issues.apache.org/jira/browse/GOBBLIN-869
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2724: [GOBBLIN-869] Update logging to include entire stack trace when adding a datanode f…
jack-moseley opened a new pull request #2724: [GOBBLIN-869] Update logging to
include entire stack trace when adding a datanode f…
URL: https://github.com/apache/incubator-gobblin/pull/2724
…ails
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-869
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-869) Update logging to include entire stack trace when adding a datanode fails
Jack Moseley created GOBBLIN-869: Summary: Update logging to include entire stack trace when adding a datanode fails Key: GOBBLIN-869 URL: https://issues.apache.org/jira/browse/GOBBLIN-869 Project: Apache Gobblin Issue Type: Bug Reporter: Jack Moseley -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] arjun4084346 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running
arjun4084346 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528644066 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-870) [GaaS] Add abfs to supported scheme
[ https://issues.apache.org/jira/browse/GOBBLIN-870?focusedWorklogId=307568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307568 ] ASF GitHub Bot logged work on GOBBLIN-870: -- Author: ASF GitHub Bot Created on: 06/Sep/19 01:43 Start Date: 06/Sep/19 01:43 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2725: [GOBBLIN-870] Adding abfs scheme URL: https://github.com/apache/incubator-gobblin/pull/2725#issuecomment-528672691 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=h1) Report > Merging [#2725](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2725 +/- ## + Coverage 45.01% 45.02% +0.01% - Complexity 8740 8742 +2 Files 1884 1884 Lines 7026670266 Branches 7709 7709 + Hits 3162831638 +10 + Misses3570935701 -8 + Partials 2929 2927 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...blin/service/modules/core/GitFlowGraphMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdEZsb3dHcmFwaE1vbml0b3IuamF2YQ==) | `73.77% <0%> (ø)` | `23 <0> (ø)` | :arrow_down: | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `66.19% <0%> (+0.46%)` | `29% <0%> (ø)` | :arrow_down: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `84.67% <0%> (+2.41%)` | `29% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=footer). Last update [3523dbc...344ad44](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307568) Time Spent: 20m (was: 10m) >
[GitHub] [incubator-gobblin] codecov-io commented on issue #2725: [GOBBLIN-870] Adding abfs scheme
codecov-io commented on issue #2725: [GOBBLIN-870] Adding abfs scheme URL: https://github.com/apache/incubator-gobblin/pull/2725#issuecomment-528672691 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=h1) Report > Merging [#2725](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2725 +/- ## + Coverage 45.01% 45.02% +0.01% - Complexity 8740 8742 +2 Files 1884 1884 Lines 7026670266 Branches 7709 7709 + Hits 3162831638 +10 + Misses3570935701 -8 + Partials 2929 2927 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...blin/service/modules/core/GitFlowGraphMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdEZsb3dHcmFwaE1vbml0b3IuamF2YQ==) | `73.77% <0%> (ø)` | `23 <0> (ø)` | :arrow_down: | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `66.19% <0%> (+0.46%)` | `29% <0%> (ø)` | :arrow_down: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `84.67% <0%> (+2.41%)` | `29% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2725/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=footer). Last update [3523dbc...344ad44](https://codecov.io/gh/apache/incubator-gobblin/pull/2725?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r321538410 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -896,6 +928,43 @@ private void reinitializeBufferedReader() throws IOException, AsyncApiException } } + private List fetchWithRetry( Review comment: Closing the thread. Please reopen it, if necessary. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=307551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307551 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 06/Sep/19 00:38 Start Date: 06/Sep/19 00:38 Worklog Time Spent: 10m Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r321538410 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -896,6 +928,43 @@ private void reinitializeBufferedReader() throws IOException, AsyncApiException } } + private List fetchWithRetry( Review comment: Closing the thread. Please reopen it, if necessary. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307551) Time Spent: 2.5h (was: 2h 20m) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 2.5h > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] linweihs opened a new pull request #2725: [GOBBLIN-870] Adding abfs scheme
linweihs opened a new pull request #2725: [GOBBLIN-870] Adding abfs scheme URL: https://github.com/apache/incubator-gobblin/pull/2725 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - https://issues.apache.org/jira/browse/GOBBLIN-870 ### Description Adding supported abfs scheme https://www.cloudera.com/documentation/enterprise/latest/topics/admin_adls2_config.html https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver Print stacktrace ### Tests trivial changes ### Commits [GOBBLIN-870] Adding abfs scheme This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running
sv2000 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528661396 @jack-moseley travis failed. can you check? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-868) Job statuses are still used for determining if a flow is running
[ https://issues.apache.org/jira/browse/GOBBLIN-868?focusedWorklogId=307554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307554 ] ASF GitHub Bot logged work on GOBBLIN-868: -- Author: ASF GitHub Bot Created on: 06/Sep/19 00:43 Start Date: 06/Sep/19 00:43 Worklog Time Spent: 10m Work Description: sv2000 commented on issue #2723: [GOBBLIN-868] Check flow status instead of job status to determine if flow is running URL: https://github.com/apache/incubator-gobblin/pull/2723#issuecomment-528661396 @jack-moseley travis failed. can you check? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307554) Time Spent: 50m (was: 40m) > Job statuses are still used for determining if a flow is running > > > Key: GOBBLIN-868 > URL: https://issues.apache.org/jira/browse/GOBBLIN-868 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-870) [GaaS] Add abfs to supported scheme
[ https://issues.apache.org/jira/browse/GOBBLIN-870?focusedWorklogId=307550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307550 ] ASF GitHub Bot logged work on GOBBLIN-870: -- Author: ASF GitHub Bot Created on: 06/Sep/19 00:37 Start Date: 06/Sep/19 00:37 Worklog Time Spent: 10m Work Description: linweihs commented on pull request #2725: [GOBBLIN-870] Adding abfs scheme URL: https://github.com/apache/incubator-gobblin/pull/2725 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - https://issues.apache.org/jira/browse/GOBBLIN-870 ### Description Adding supported abfs scheme https://www.cloudera.com/documentation/enterprise/latest/topics/admin_adls2_config.html https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver Print stacktrace ### Tests trivial changes ### Commits [GOBBLIN-870] Adding abfs scheme This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 307550) Remaining Estimate: 0h Time Spent: 10m > [GaaS] Add abfs to supported scheme > --- > > Key: GOBBLIN-870 > URL: https://issues.apache.org/jira/browse/GOBBLIN-870 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Wei-Hsiang Lin >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Adding Azure datalake Gen2 scheme > abfs[s]://@.dfs.core.windows.net// > [https://www.cloudera.com/documentation/enterprise/latest/topics/admin_adls2_config.html] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (GOBBLIN-870) [GaaS] Add abfs to supported scheme
Wei-Hsiang Lin created GOBBLIN-870: -- Summary: [GaaS] Add abfs to supported scheme Key: GOBBLIN-870 URL: https://issues.apache.org/jira/browse/GOBBLIN-870 Project: Apache Gobblin Issue Type: Bug Reporter: Wei-Hsiang Lin Adding Azure datalake Gen2 scheme abfs[s]://@.dfs.core.windows.net// [https://www.cloudera.com/documentation/enterprise/latest/topics/admin_adls2_config.html] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r321539964
##
File path:
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
##
@@ -146,6 +156,95 @@ protected void addLineageSourceInfo(SourceState
sourceState, SourceEntity entity
@Override
protected List generateWorkUnits(SourceEntity sourceEntity,
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "PK_CHUNKING");
+if (partitionType.equals("PK_CHUNKING")) {
+ return generateWorkUnitsPkChunking(sourceEntity, state,
previousWatermark);
+} else {
+ return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark);
+}
+ }
+
+ /**
+ * generate workUnit with noQuery=true
+ */
+ private List generateWorkUnitsPkChunking(SourceEntity
sourceEntity, SourceState state, long previousWatermark) {
+ List batchIdAndResultIds =
executeQueryWithPkChunking(state, previousWatermark);
+ List ret = createWorkUnits(sourceEntity, state,
batchIdAndResultIds);
+ return ret;
+ }
+
+ private List
executeQueryWithPkChunking(
+ SourceState sourceState,
+ long previousWatermark
+ ) throws RuntimeException {
+Properties commonProperties = sourceState.getCommonProperties();
+Properties specProperties = sourceState.getSpecProperties();
+State state = new State();
+state.setProps(commonProperties, specProperties);
+WorkUnit workUnit = WorkUnit.createEmpty();
+try {
+ WorkUnitState workUnitState = new WorkUnitState(workUnit, state);
+ workUnitState.setId("test" + new Random().nextInt());
+ workUnitState.setProp(ENABLE_PK_CHUNKING_KEY, true); // set extractor
enable pk chunking
+ int chunkSize = workUnitState.getPropAsInt(PARTITION_PK_CHUNKING_SIZE,
DEFAULT_PK_CHUNKING_SIZE);
+ workUnitState.setProp(PK_CHUNKING_SIZE_KEY, chunkSize); // set extractor
pk chunking size
+ workUnitState.setProp(PK_CHUNKING_SKIP_COUNT_CHECK, true); // don't use
count check for we couldn't get count
+ SalesforceExtractor salesforceExtractor = (SalesforceExtractor)
this.getExtractor(workUnitState);
+ String entity = sourceState.getProp(ConfigurationKeys.SOURCE_ENTITY);
+ Partitioner partitioner = new Partitioner(sourceState);
+ if (isEarlyStopEnabled(state) && partitioner.isFullDump()) {
+throw new UnsupportedOperationException("Early stop mode cannot work
with full dump mode.");
+ }
+ Partition partition = partitioner.getGlobalPartition(previousWatermark);
+ String condition = "";
Review comment:
How about we leave this refactoring later?
It doesn't look like I can use the function directly. In PK-chunking case,
we only have start-time, we don't have end-time. end-time is always the
current-time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=307556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307556
]
ASF GitHub Bot logged work on GOBBLIN-865:
--
Author: ASF GitHub Bot
Created on: 06/Sep/19 00:49
Start Date: 06/Sep/19 00:49
Worklog Time Spent: 10m
Work Description: arekusuri commented on pull request #2722: GOBBLIN-865:
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r321539964
##
File path:
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
##
@@ -146,6 +156,95 @@ protected void addLineageSourceInfo(SourceState
sourceState, SourceEntity entity
@Override
protected List generateWorkUnits(SourceEntity sourceEntity,
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "PK_CHUNKING");
+if (partitionType.equals("PK_CHUNKING")) {
+ return generateWorkUnitsPkChunking(sourceEntity, state,
previousWatermark);
+} else {
+ return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark);
+}
+ }
+
+ /**
+ * generate workUnit with noQuery=true
+ */
+ private List generateWorkUnitsPkChunking(SourceEntity
sourceEntity, SourceState state, long previousWatermark) {
+ List batchIdAndResultIds =
executeQueryWithPkChunking(state, previousWatermark);
+ List ret = createWorkUnits(sourceEntity, state,
batchIdAndResultIds);
+ return ret;
+ }
+
+ private List
executeQueryWithPkChunking(
+ SourceState sourceState,
+ long previousWatermark
+ ) throws RuntimeException {
+Properties commonProperties = sourceState.getCommonProperties();
+Properties specProperties = sourceState.getSpecProperties();
+State state = new State();
+state.setProps(commonProperties, specProperties);
+WorkUnit workUnit = WorkUnit.createEmpty();
+try {
+ WorkUnitState workUnitState = new WorkUnitState(workUnit, state);
+ workUnitState.setId("test" + new Random().nextInt());
+ workUnitState.setProp(ENABLE_PK_CHUNKING_KEY, true); // set extractor
enable pk chunking
+ int chunkSize = workUnitState.getPropAsInt(PARTITION_PK_CHUNKING_SIZE,
DEFAULT_PK_CHUNKING_SIZE);
+ workUnitState.setProp(PK_CHUNKING_SIZE_KEY, chunkSize); // set extractor
pk chunking size
+ workUnitState.setProp(PK_CHUNKING_SKIP_COUNT_CHECK, true); // don't use
count check for we couldn't get count
+ SalesforceExtractor salesforceExtractor = (SalesforceExtractor)
this.getExtractor(workUnitState);
+ String entity = sourceState.getProp(ConfigurationKeys.SOURCE_ENTITY);
+ Partitioner partitioner = new Partitioner(sourceState);
+ if (isEarlyStopEnabled(state) && partitioner.isFullDump()) {
+throw new UnsupportedOperationException("Early stop mode cannot work
with full dump mode.");
+ }
+ Partition partition = partitioner.getGlobalPartition(previousWatermark);
+ String condition = "";
Review comment:
How about we leave this refactoring later?
It doesn't look like I can use the function directly. In PK-chunking case,
we only have start-time, we don't have end-time. end-time is always the
current-time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307556)
Time Spent: 2h 40m (was: 2.5h)
> Add feature that enables PK-chunking in partition
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Alex Li
>Priority: Major
> Labels: salesforce
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a
> giant query to multiple sub queries. There are 3 mechanisms:
> * simple partition (equally split by time)
> * dynamic pre-partition (generate histogram and split by row numbers)
> * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>
> The pk-chunking doc from SFDC -
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [incubator-gobblin] codecov-io commented on issue #2724: [GOBBLIN-869] Update logging to include entire stack trace when adding a datanode f…
codecov-io commented on issue #2724: [GOBBLIN-869] Update logging to include entire stack trace when adding a datanode f… URL: https://github.com/apache/incubator-gobblin/pull/2724#issuecomment-528665166 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=h1) Report > Merging [#2724](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2724 +/- ## + Coverage 45.01% 45.01% +<.01% - Complexity 8740 8741 +1 Files 1884 1884 Lines 7026670266 Branches 7709 7709 + Hits 3162831632 +4 + Misses3570935703 -6 - Partials 2929 2931 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...blin/service/modules/core/GitFlowGraphMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdEZsb3dHcmFwaE1vbml0b3IuamF2YQ==) | `73.77% <0%> (ø)` | `23 <0> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0%> (-1.43%)` | `4% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.78% <0%> (-0.94%)` | `29% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.53% <0%> (+0.33%)` | `24% <0%> (ø)` | :arrow_down: | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=) | `68.6% <0%> (+2.32%)` | `11% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0%> (+7.14%)` | `3% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=footer). Last update [3523dbc...bd643f1](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please
[jira] [Work logged] (GOBBLIN-869) Update logging to include entire stack trace when adding a datanode fails
[ https://issues.apache.org/jira/browse/GOBBLIN-869?focusedWorklogId=307562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307562 ] ASF GitHub Bot logged work on GOBBLIN-869: -- Author: ASF GitHub Bot Created on: 06/Sep/19 01:04 Start Date: 06/Sep/19 01:04 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2724: [GOBBLIN-869] Update logging to include entire stack trace when adding a datanode f… URL: https://github.com/apache/incubator-gobblin/pull/2724#issuecomment-528665166 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=h1) Report > Merging [#2724](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/3523dbcdf41d489ece761c37f5849539cfc8948d?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2724 +/- ## + Coverage 45.01% 45.01% +<.01% - Complexity 8740 8741 +1 Files 1884 1884 Lines 7026670266 Branches 7709 7709 + Hits 3162831632 +4 + Misses3570935703 -6 - Partials 2929 2931 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...blin/service/modules/core/GitFlowGraphMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdEZsb3dHcmFwaE1vbml0b3IuamF2YQ==) | `73.77% <0%> (ø)` | `23 <0> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0%> (-1.43%)` | `4% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.78% <0%> (-0.94%)` | `29% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.53% <0%> (+0.33%)` | `24% <0%> (ø)` | :arrow_down: | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=) | `68.6% <0%> (+2.32%)` | `11% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2724/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0%> (+7.14%)` | `3% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2724?src=pr=footer). Last update
