date:20190913

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312459
 ]

ASF GitHub Bot logged work on GOBBLIN-881:
--

Author: ASF GitHub Bot
Created on: 14/Sep/19 00:58
Start Date: 14/Sep/19 00:58
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2735: [GOBBLIN-881] Add 
job tag field that can be used to filter job statuses
URL: 
https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531431862
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=h1)
 Report
   > Merging 
[#2735](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `30.76%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2735  +/-   ##
   
   + Coverage 45.04%   45.05%   +0.01% 
   - Complexity 8751 8752   +1 
   
 Files  1884 1884  
 Lines 7031170318   +7 
 Branches   7716 7718   +2 
   
   + Hits  3167231685  +13 
   + Misses3569735692   -5 
   + Partials   2942 2941   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../org/apache/gobblin/metrics/event/TimingEvent.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9UaW1pbmdFdmVudC5qYXZh)
 | `70% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: |
   | 
[...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==)
 | `22.22% <0%> (-1.31%)` | `4 <0> (ø)` | |
   | 
[...ervice/modules/orchestration/TimingEventUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL1RpbWluZ0V2ZW50VXRpbHMuamF2YQ==)
 | `95.65% <100%> (+0.19%)` | `4 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=)
 | `72.34% <100%> (ø)` | `7 <0> (ø)` | :arrow_down: |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `74.19% <40%> (-7.95%)` | `11 <4> (ø)` | |
   | 
[...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=)
 | `66.27% <0%> (-2.33%)` | `11% <0%>

[GitHub] [incubator-gobblin] codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

2019-09-13 Thread GitBox

codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can 
be used to filter job statuses
URL: 
https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531431862
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=h1)
 Report
   > Merging 
[#2735](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `30.76%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2735  +/-   ##
   
   + Coverage 45.04%   45.05%   +0.01% 
   - Complexity 8751 8752   +1 
   
 Files  1884 1884  
 Lines 7031170318   +7 
 Branches   7716 7718   +2 
   
   + Hits  3167231685  +13 
   + Misses3569735692   -5 
   + Partials   2942 2941   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../org/apache/gobblin/metrics/event/TimingEvent.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9UaW1pbmdFdmVudC5qYXZh)
 | `70% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: |
   | 
[...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==)
 | `22.22% <0%> (-1.31%)` | `4 <0> (ø)` | |
   | 
[...ervice/modules/orchestration/TimingEventUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL1RpbWluZ0V2ZW50VXRpbHMuamF2YQ==)
 | `95.65% <100%> (+0.19%)` | `4 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=)
 | `72.34% <100%> (ø)` | `7 <0> (ø)` | :arrow_down: |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `74.19% <40%> (-7.95%)` | `11 <4> (ø)` | |
   | 
[...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=)
 | `66.27% <0%> (-2.33%)` | `11% <0%> (ø)` | |
   | 
[.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh)
 | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | |
   | ... and [6

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312450
 ]

ASF GitHub Bot logged work on GOBBLIN-881:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 23:53
Start Date: 13/Sep/19 23:53
Worklog Time Spent: 10m 
  Work Description: jack-moseley commented on pull request #2735: 
[GOBBLIN-881] Add job tag field that can be used to filter job statuses
URL: https://github.com/apache/incubator-gobblin/pull/2735
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-881
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   
   With multi-dataset support in gaas, it is more convenient to have an extra 
field that can be used to filter when querying the job status (this would be 
dataset name in most cases).
   
   This PR allows jobs to include `job.tag` field then when querying jobStatus, 
specifying tag in the query will return only job statuses with that tag.
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Updated test
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312450)
Remaining Estimate: 0h
Time Spent: 10m

> Add job tag field that can be used to filter job statuses
> -
>
> Key: GOBBLIN-881
> URL: https://issues.apache.org/jira/browse/GOBBLIN-881
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Jack Moseley
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312451
 ]

ASF GitHub Bot logged work on GOBBLIN-881:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 23:53
Start Date: 13/Sep/19 23:53
Worklog Time Spent: 10m 
  Work Description: jack-moseley commented on issue #2735: [GOBBLIN-881] 
Add job tag field that can be used to filter job statuses
URL: 
https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531424495
 
 
   @arjun4084346 please review
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312451)
Time Spent: 20m  (was: 10m)

> Add job tag field that can be used to filter job statuses
> -
>
> Key: GOBBLIN-881
> URL: https://issues.apache.org/jira/browse/GOBBLIN-881
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Jack Moseley
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [incubator-gobblin] jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

2019-09-13 Thread GitBox

jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can 
be used to filter job statuses
URL: 
https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531424495
 
 
   @arjun4084346 please review


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

2019-09-13 Thread GitBox

jack-moseley opened a new pull request #2735: [GOBBLIN-881] Add job tag field 
that can be used to filter job statuses
URL: https://github.com/apache/incubator-gobblin/pull/2735
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-881
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   
   With multi-dataset support in gaas, it is more convenient to have an extra 
field that can be used to filter when querying the job status (this would be 
dataset name in most cases).
   
   This PR allows jobs to include `job.tag` field then when querying jobStatus, 
specifying tag in the query will return only job statuses with that tag.
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Updated test
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

2019-09-13 Thread Jack Moseley (Jira)

Jack Moseley created GOBBLIN-881:


 Summary: Add job tag field that can be used to filter job statuses
 Key: GOBBLIN-881
 URL: https://issues.apache.org/jira/browse/GOBBLIN-881
 Project: Apache Gobblin
  Issue Type: Bug
Reporter: Jack Moseley






--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312446
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 23:45
Start Date: 13/Sep/19 23:45
Worklog Time Spent: 10m 
  Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: 
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324393059
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,101 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  // pk-chunking only supports start-time by 
source.querybased.start.value, and does not support end-time.
+  // always ingest data later than or equal source.querybased.start.value.
+  // we should only pk chunking based work units only in case of 
snapshot/full ingestion
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
+} else {
+  return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark);
+}
+  }
+
+  /**
+   * generate workUnit with noQuery=true
+   */
+  private List generateWorkUnitsPkChunking(SourceEntity 
sourceEntity, SourceState state, long previousWatermark) {
+  SalesforceBulkJobId salesforceBulkJobId = 
executeQueryWithPkChunking(state, previousWatermark);
+  List ret = createWorkUnits(sourceEntity, state, 
salesforceBulkJobId);
+  return ret;
+  }
+
+  private SalesforceBulkJobId executeQueryWithPkChunking(
+  SourceState sourceState,
+  long previousWatermark
+  ) throws RuntimeException {
+State state = new State(sourceState);
+WorkUnit workUnit = WorkUnit.createEmpty();
+try {
+  WorkUnitState workUnitState = new WorkUnitState(workUnit, state);
+  workUnitState.setId("Execute pk-chunking");
 
 Review comment:
   Hi @zxcware 
   is this OK? I am trying to set id for workUnit.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312446)
Time Spent: 4h 50m  (was: 4h 40m)

> Add feature that enables PK-chunking in partition 
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Alex Li
>Priority: Major
>  Labels: salesforce
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a 
> giant query to multiple sub queries. There are 3 mechanisms:
>  * simple partition (equally split by time)
>  * dynamic pre-partition (generate histogram and split by row numbers)
>  * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to 
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>  
> The pk-chunking doc from SFDC - 
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature 
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324393059
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,101 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  // pk-chunking only supports start-time by 
source.querybased.start.value, and does not support end-time.
+  // always ingest data later than or equal source.querybased.start.value.
+  // we should only pk chunking based work units only in case of 
snapshot/full ingestion
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
+} else {
+  return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark);
+}
+  }
+
+  /**
+   * generate workUnit with noQuery=true
+   */
+  private List generateWorkUnitsPkChunking(SourceEntity 
sourceEntity, SourceState state, long previousWatermark) {
+  SalesforceBulkJobId salesforceBulkJobId = 
executeQueryWithPkChunking(state, previousWatermark);
+  List ret = createWorkUnits(sourceEntity, state, 
salesforceBulkJobId);
+  return ret;
+  }
+
+  private SalesforceBulkJobId executeQueryWithPkChunking(
+  SourceState sourceState,
+  long previousWatermark
+  ) throws RuntimeException {
+State state = new State(sourceState);
+WorkUnit workUnit = WorkUnit.createEmpty();
+try {
+  WorkUnitState workUnitState = new WorkUnitState(workUnit, state);
+  workUnitState.setId("Execute pk-chunking");
 
 Review comment:
   Hi @zxcware 
   is this OK? I am trying to set id for workUnit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312419
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 22:53
Start Date: 13/Sep/19 22:53
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2722: GOBBLIN-865: Add 
feature that enables PK-chunking in partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=h1)
 Report
   > Merging 
[#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr=desc)
 will **decrease** coverage by `0.08%`.
   > The diff coverage is `1.78%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2722  +/-   ##
   
   - Coverage 45.04%   44.96%   -0.09% 
   - Complexity 8739 8752  +13 
   
 Files  1880 1884   +4 
 Lines 7020570454 +249 
 Branches   7707 7730  +23 
   
   + Hits  3162331678  +55 
   - Misses3565135831 +180 
   - Partials   2931 2945  +14
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | |
   | 
[...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh)
 | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | |
   | 
[...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh)
 | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   |

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that 
enables PK-chunking in partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=h1)
 Report
   > Merging 
[#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr=desc)
 will **decrease** coverage by `0.08%`.
   > The diff coverage is `1.78%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2722  +/-   ##
   
   - Coverage 45.04%   44.96%   -0.09% 
   - Complexity 8739 8752  +13 
   
 Files  1880 1884   +4 
 Lines 7020570454 +249 
 Branches   7707 7730  +23 
   
   + Hits  3162331678  +55 
   - Misses3565135831 +180 
   - Partials   2931 2945  +14
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | |
   | 
[...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh)
 | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | |
   | 
[...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh)
 | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | 
[...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | ... and [18 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312402
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 22:17
Start Date: 13/Sep/19 22:17
Worklog Time Spent: 10m 
  Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: 
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385831
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
 
 Review comment:
   Thanks, will add it!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312402)
Time Spent: 4.5h  (was: 4h 20m)

> Add feature that enables PK-chunking in partition 
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Alex Li
>Priority: Major
>  Labels: salesforce
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a 
> giant query to multiple sub queries. There are 3 mechanisms:
>  * simple partition (equally split by time)
>  * dynamic pre-partition (generate histogram and split by row numbers)
>  * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to 
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>  
> The pk-chunking doc from SFDC - 
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature 
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385831
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
 
 Review comment:
   Thanks, will add it!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312400
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 22:13
Start Date: 13/Sep/19 22:13
Worklog Time Spent: 10m 
  Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: 
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385070
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java
 ##
 @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String 
column, long value, String v
 return dataTypeMap;
   }
 
+
+  private String partitionPkChunkingJobId = null;
+  private Iterator partitionPkChunkingBatchIdResultIterator = null;
+
+  private Iterator getRecordSetPkchunking(WorkUnit workUnit) 
throws RuntimeException {
+if (partitionPkChunkingBatchIdResultIterator == null) {
+  partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID);
+  partitionPkChunkingBatchIdResultIterator = 
Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator();
+}
+if (!partitionPkChunkingBatchIdResultIterator.hasNext()) {
+  return null;
+}
+try {
+  if (!bulkApiLogin()) {
+throw new IllegalArgumentException("Invalid Login");
+  }
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+String[] batchIdResultIdArray = 
partitionPkChunkingBatchIdResultIterator.next().split(":");
+String batchId = batchIdResultIdArray[0];
+String resultId = batchIdResultIdArray[1];
+List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, 
partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit);
+return rs.iterator();
+  }
+
   @Override
   public Iterator getRecordSetFromSourceApi(String schema, String 
entity, WorkUnit workUnit,
   List predicateList) throws IOException {
 log.debug("Getting salesforce data using bulk api");
-RecordSet rs = null;
+
+// new version of extractor: bulk api with pk-chunking in pre-partitioning 
of SalesforceSource
+if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) {
 
 Review comment:
   Thanks! will do.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312400)
Time Spent: 4h 20m  (was: 4h 10m)

> Add feature that enables PK-chunking in partition 
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Alex Li
>Priority: Major
>  Labels: salesforce
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a 
> giant query to multiple sub queries. There are 3 mechanisms:
>  * simple partition (equally split by time)
>  * dynamic pre-partition (generate histogram and split by row numbers)
>  * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to 
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>  
> The pk-chunking doc from SFDC - 
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature 
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385070
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java
 ##
 @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String 
column, long value, String v
 return dataTypeMap;
   }
 
+
+  private String partitionPkChunkingJobId = null;
+  private Iterator partitionPkChunkingBatchIdResultIterator = null;
+
+  private Iterator getRecordSetPkchunking(WorkUnit workUnit) 
throws RuntimeException {
+if (partitionPkChunkingBatchIdResultIterator == null) {
+  partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID);
+  partitionPkChunkingBatchIdResultIterator = 
Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator();
+}
+if (!partitionPkChunkingBatchIdResultIterator.hasNext()) {
+  return null;
+}
+try {
+  if (!bulkApiLogin()) {
+throw new IllegalArgumentException("Invalid Login");
+  }
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+String[] batchIdResultIdArray = 
partitionPkChunkingBatchIdResultIterator.next().split(":");
+String batchId = batchIdResultIdArray[0];
+String resultId = batchIdResultIdArray[1];
+List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, 
partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit);
+return rs.iterator();
+  }
+
   @Override
   public Iterator getRecordSetFromSourceApi(String schema, String 
entity, WorkUnit workUnit,
   List predicateList) throws IOException {
 log.debug("Getting salesforce data using bulk api");
-RecordSet rs = null;
+
+// new version of extractor: bulk api with pk-chunking in pre-partitioning 
of SalesforceSource
+if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) {
 
 Review comment:
   Thanks! will do.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312399
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 22:09
Start Date: 13/Sep/19 22:09
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2722: GOBBLIN-865: 
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324384004
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
 
 Review comment:
   This is where to comment the right use cases to enable `PK_CHUNKING`, as 
it's the entry point
   ```
 // pk-chunking only supports start-time by 
source.querybased.start.value, and does not support end-time.
 // always ingest data later than or equal 
source.querybased.start.value.
 // we should only pk chunking based work units only in case of 
snapshot/full ingestion
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312399)
Time Spent: 4h 10m  (was: 4h)

> Add feature that enables PK-chunking in partition 
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Alex Li
>Priority: Major
>  Labels: salesforce
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a 
> giant query to multiple sub queries. There are 3 mechanisms:
>  * simple partition (equally split by time)
>  * dynamic pre-partition (generate histogram and split by row numbers)
>  * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to 
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>  
> The pk-chunking doc from SFDC - 
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312398
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 22:09
Start Date: 13/Sep/19 22:09
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2722: GOBBLIN-865: 
Add feature that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324383638
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java
 ##
 @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String 
column, long value, String v
 return dataTypeMap;
   }
 
+
+  private String partitionPkChunkingJobId = null;
+  private Iterator partitionPkChunkingBatchIdResultIterator = null;
+
+  private Iterator getRecordSetPkchunking(WorkUnit workUnit) 
throws RuntimeException {
+if (partitionPkChunkingBatchIdResultIterator == null) {
+  partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID);
+  partitionPkChunkingBatchIdResultIterator = 
Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator();
+}
+if (!partitionPkChunkingBatchIdResultIterator.hasNext()) {
+  return null;
+}
+try {
+  if (!bulkApiLogin()) {
+throw new IllegalArgumentException("Invalid Login");
+  }
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+String[] batchIdResultIdArray = 
partitionPkChunkingBatchIdResultIterator.next().split(":");
+String batchId = batchIdResultIdArray[0];
+String resultId = batchIdResultIdArray[1];
+List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, 
partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit);
+return rs.iterator();
+  }
+
   @Override
   public Iterator getRecordSetFromSourceApi(String schema, String 
entity, WorkUnit workUnit,
   List predicateList) throws IOException {
 log.debug("Getting salesforce data using bulk api");
-RecordSet rs = null;
+
+// new version of extractor: bulk api with pk-chunking in pre-partitioning 
of SalesforceSource
+if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) {
 
 Review comment:
   Are you checking if `PK_CHUNKING_JOB_ID` exists in workUnit? If so, we can 
use `workUnit.contains(PK_CHUNKING_JOB_ID)`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312398)
Time Spent: 4h 10m  (was: 4h)

> Add feature that enables PK-chunking in partition 
> --
>
> Key: GOBBLIN-865
> URL: https://issues.apache.org/jira/browse/GOBBLIN-865
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Alex Li
>Priority: Major
>  Labels: salesforce
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> In SFDC(salesforce) connector, we have partitioning mechanisms to split a 
> giant query to multiple sub queries. There are 3 mechanisms:
>  * simple partition (equally split by time)
>  * dynamic pre-partition (generate histogram and split by row numbers)
>  * user specified partition (set up time range in job file)
> However there are tables like Task and Contract are failing time to time to 
> fetch full data.
> We may want to utilize PK-chunking to partition the query.
>  
> The pk-chunking doc from SFDC - 
> [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature 
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324384004
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
 ##
 @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState 
sourceState, SourceEntity entity
 
   @Override
   protected List generateWorkUnits(SourceEntity sourceEntity, 
SourceState state, long previousWatermark) {
+String partitionType = state.getProp(PARTITION_TYPE, "");
+if (partitionType.equals("PK_CHUNKING")) {
+  return generateWorkUnitsPkChunking(sourceEntity, state, 
previousWatermark);
 
 Review comment:
   This is where to comment the right use cases to enable `PK_CHUNKING`, as 
it's the entry point
   ```
 // pk-chunking only supports start-time by 
source.querybased.start.value, and does not support end-time.
 // always ingest data later than or equal 
source.querybased.start.value.
 // we should only pk chunking based work units only in case of 
snapshot/full ingestion
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature 
that enables PK-chunking in partition
URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324383638
 
 

 ##
 File path: 
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java
 ##
 @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String 
column, long value, String v
 return dataTypeMap;
   }
 
+
+  private String partitionPkChunkingJobId = null;
+  private Iterator partitionPkChunkingBatchIdResultIterator = null;
+
+  private Iterator getRecordSetPkchunking(WorkUnit workUnit) 
throws RuntimeException {
+if (partitionPkChunkingBatchIdResultIterator == null) {
+  partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID);
+  partitionPkChunkingBatchIdResultIterator = 
Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator();
+}
+if (!partitionPkChunkingBatchIdResultIterator.hasNext()) {
+  return null;
+}
+try {
+  if (!bulkApiLogin()) {
+throw new IllegalArgumentException("Invalid Login");
+  }
+} catch (Exception e) {
+  throw new RuntimeException(e);
+}
+String[] batchIdResultIdArray = 
partitionPkChunkingBatchIdResultIterator.next().split(":");
+String batchId = batchIdResultIdArray[0];
+String resultId = batchIdResultIdArray[1];
+List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, 
partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit);
+return rs.iterator();
+  }
+
   @Override
   public Iterator getRecordSetFromSourceApi(String schema, String 
entity, WorkUnit workUnit,
   List predicateList) throws IOException {
 log.debug("Getting salesforce data using bulk api");
-RecordSet rs = null;
+
+// new version of extractor: bulk api with pk-chunking in pre-partitioning 
of SalesforceSource
+if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) {
 
 Review comment:
   Are you checking if `PK_CHUNKING_JOB_ID` exists in workUnit? If so, we can 
use `workUnit.contains(PK_CHUNKING_JOB_ID)`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] zxcware commented on issue #2733: Modify GaaS configuration so that it runs through the startup script

2019-09-13 Thread GitBox

zxcware commented on issue #2733: Modify GaaS configuration so that it runs 
through the startup script
URL: 
https://github.com/apache/incubator-gobblin/pull/2733#issuecomment-531402997
 
 
   Hi @sv2000 , please review and merge the pr.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-880?focusedWorklogId=312362=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312362
 ]

ASF GitHub Bot logged work on GOBBLIN-880:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 21:47
Start Date: 13/Sep/19 21:47
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2734: GOBBLIN-880 Bump 
CouchbaseWriter Couchbase SDK version + write docs +…
URL: 
https://github.com/apache/incubator-gobblin/pull/2734#issuecomment-531401555
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=h1)
 Report
   > Merging 
[#2734](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `59.4%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2734  +/-   ##
   
   + Coverage 45.04%   45.07%   +0.03% 
   - Complexity 8751 8761  +10 
   
 Files  1884 1884  
 Lines 7031170379  +68 
 Branches   7716 7723   +7 
   
   + Hits  3167231726  +54 
   - Misses3569735708  +11 
   - Partials   2942 2945   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/couchbase/writer/CouchbaseWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQnVpbGRlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../couchbase/writer/CouchbaseEnvironmentFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlRW52aXJvbm1lbnRGYWN0b3J5LmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==)
 | `95.65% <100%> (+1.9%)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==)
 | `60.4% <100%> (+2.54%)` | `40 <3> (+3)` | :arrow_up: |
   | 
[...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=)
 | `64.39% <61.97%> (-4.22%)` | `15 <4> (+4)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `65.72% <0%> (+1.4%)` | `28% <0%> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `39.25% <0%> (+6.54%)` | `13% <0%> (+2%)` | :arrow_up: |
   
   --

[GitHub] [incubator-gobblin] codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…

2019-09-13 Thread GitBox

codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase 
SDK version + write docs +…
URL: 
https://github.com/apache/incubator-gobblin/pull/2734#issuecomment-531401555
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=h1)
 Report
   > Merging 
[#2734](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `59.4%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2734  +/-   ##
   
   + Coverage 45.04%   45.07%   +0.03% 
   - Complexity 8751 8761  +10 
   
 Files  1884 1884  
 Lines 7031170379  +68 
 Branches   7716 7723   +7 
   
   + Hits  3167231726  +54 
   - Misses3569735708  +11 
   - Partials   2942 2945   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/couchbase/writer/CouchbaseWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQnVpbGRlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../couchbase/writer/CouchbaseEnvironmentFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlRW52aXJvbm1lbnRGYWN0b3J5LmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==)
 | `95.65% <100%> (+1.9%)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==)
 | `60.4% <100%> (+2.54%)` | `40 <3> (+3)` | :arrow_up: |
   | 
[...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=)
 | `64.39% <61.97%> (-4.22%)` | `15 <4> (+4)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `65.72% <0%> (+1.4%)` | `28% <0%> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `39.25% <0%> (+6.54%)` | `13% <0%> (+2%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr=footer).
 Last update

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312359
 ]

ASF GitHub Bot logged work on GOBBLIN-865:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 21:44
Start Date: 13/Sep/19 21:44
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2722: GOBBLIN-865: Add 
feature that enables PK-chunking in partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=h1)
 Report
   > Merging 
[#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr=desc)
 will **decrease** coverage by `0.07%`.
   > The diff coverage is `1.78%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2722  +/-   ##
   
   - Coverage 45.04%   44.96%   -0.08% 
   - Complexity 8739 8753  +14 
   
 Files  1880 1884   +4 
 Lines 7020570454 +249 
 Branches   7707 7730  +23 
   
   + Hits  3162331680  +57 
   - Misses3565135831 +180 
   - Partials   2931 2943  +12
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | |
   | 
[...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh)
 | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | |
   | 
[...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh)
 | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   |

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

2019-09-13 Thread GitBox

codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that 
enables PK-chunking in partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=h1)
 Report
   > Merging 
[#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr=desc)
 will **decrease** coverage by `0.07%`.
   > The diff coverage is `1.78%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2722  +/-   ##
   
   - Coverage 45.04%   44.96%   -0.08% 
   - Complexity 8739 8753  +14 
   
 Files  1880 1884   +4 
 Lines 7020570454 +249 
 Branches   7707 7730  +23 
   
   + Hits  3162331680  +57 
   - Misses3565135831 +180 
   - Partials   2931 2943  +12
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | |
   | 
[...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=)
 | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | |
   | 
[...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh)
 | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | |
   | 
[...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh)
 | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | 
[...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==)
 | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | ... and [20 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at

[jira] [Commented] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

2019-09-13 Thread Michael A Menarguez (Jira)



[ 
https://issues.apache.org/jira/browse/GOBBLIN-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929538#comment-16929538
 ] 

Michael A Menarguez commented on GOBBLIN-880:
-

Submited PR [https://github.com/apache/incubator-gobblin/pull/2734]

> Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + 
> enable TTL + dnsSrv
> ---
>
> Key: GOBBLIN-880
> URL: https://issues.apache.org/jira/browse/GOBBLIN-880
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-couchbase
>Reporter: Michael A Menarguez
>Assignee: Shirshanka Das
>Priority: Major
>  Labels: Couchbase
>   Original Estimate: 168h
>  Time Spent: 10m
>  Remaining Estimate: 167h 50m
>
> h1. h1. CURRENT ISSUES
> Currently CouchbaseWriter.java lacks the ability to do the following:
>  # Use certificate based authentication
>  # Set document expiry (TTL)
>  ** based on write time
>  ** based on an offset specified field contained in the record's data (JSON)
>  ** (WILL NOT ADRESS) set expiry based on a field contained in the record's 
> data
>  # Set DNS SRV for bootstrap host discovery setting
>  # Missing documentation on CouchbaseWriter usage
>  # Testing does not bring in CouchbaseMock correctly and causes problems 
> while bumping com.couchbase.client:java-client
> h1. h1. PROPOSED SOLUTIONS
>  # Add logic to connect using certificate based auth to the buckets (Will 
> need to bump  com.couchbase.client:java-client to a newer version like 2.7.6) 
> and associated configs
>  # TTL implementation
>  ## Add configs to allow setting a TTL (documentTTL) and also specify the 
> timeunits (documentTTLUnits) of these settings
>  ## Add logic to specify the path to key to the field containing the source 
> timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to 
> disambiguate between UNIX (sec) timestamps and other formats like timestamps 
> in milliseconds.
>  ## N/A but logic would be similar to (2)
>  # Add missing dnsSrv config
>  # Write proper documentation
>  # Bring in CouchbaseMock from Gradle and adapt existing unit tests.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-880?focusedWorklogId=312348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312348
 ]

ASF GitHub Bot logged work on GOBBLIN-880:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 21:14
Start Date: 13/Sep/19 21:14
Worklog Time Spent: 10m 
  Work Description: menarguez commented on pull request #2734: GOBBLIN-880 
Bump CouchbaseWriter Couchbase SDK version + write docs +…
URL: https://github.com/apache/incubator-gobblin/pull/2734
 
 
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-880
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if 
applicable):
Plese read https://issues.apache.org/jira/browse/GOBBLIN-880
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
Added unit tests and tested to push data to a couchbase cluster though an 
azkaban job using hadoopJava and a snapshot of the library. Added log lines to 
verify expiration times are set correctly and tested pushing using cert based 
authentication. Please take a look at the following obfuscated data:
   
   `13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - 
INFO Setting ttl timestamp 1569234590 for document 'key1' and data 
{"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"pvFbXpmKT7ytUn8WYBic8A==","time":1566642590043}
   13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - 
INFO Setting ttl timestamp 1569013770 for document 'key2==' and data 
{"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"CtzzEq1rQ4qLXMT8zTD89w==","time":1566421770972}
   `
   and the records are on the bucket:
   `
   cbshell [ei-ltx1/lss-search-sales-seat-lead-activity]> get 
121084037038108585_91708167944609950_506_CtzzEq1rQ4qLXMT8zTD89w==
   {
 "action": "xxx",
 "memberId": xxx,
 "requestId": "CtzzEq1rQ4qLXMT8zTD89w==",
 "targetId": xxx,
 "time": 1566421770972
   }
   `
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…
   
   … cert based auth + enable TTL + dnsSrv
   
   JIRA ticket: https://issues.apache.org/jira/browse/GOBBLIN-880
   RB Changes:
   1 - Added logic to connect using certificate based auth to the buckets 
(Will need to bump  com.couchbase.client:java-client to a newer version like 
2.7.6) and associated configs
   2 - TTL  implementation
 * Added configs to allow setting a TTL (documentTTL) and also specify 
the timeunits (documentTTLUnits) of these settings
 * Added logic to specify the path to key to the field containing the 
source timestamp (documentTTLOriginField) and its units 
(documentTTLOriginUnits) to disambiguate between UNIX (sec) timestamps and 
other formats like timestamps in milliseconds.
   3 - Added missing dnsSrv config
   4 - Written proper documentation on 
gobblin-docs/writers/CouchbaseWriter.md
   5 - Brought in CouchbaseMock from Gradle and adapt existing unit tests.
   6 - Added getTimeUnit to ConfigUtils + Unit tests
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 312348)
Remaining Estimate: 167h 50m  (was: 168h)
Time Spent: 10m

> Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + 
> enable TTL + dnsSrv
> ---
>
> Key: GOBBLIN-880
> URL: https://issues.apache.org/jira/browse/GOBBLIN-880
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-couchbase
>Reporter: Michael A Menarguez
>Assignee: Shirshanka Das
>

[GitHub] [incubator-gobblin] menarguez opened a new pull request #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…

2019-09-13 Thread GitBox

menarguez opened a new pull request #2734: GOBBLIN-880 Bump CouchbaseWriter 
Couchbase SDK version + write docs +…
URL: https://github.com/apache/incubator-gobblin/pull/2734
 
 
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-880
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if 
applicable):
Plese read https://issues.apache.org/jira/browse/GOBBLIN-880
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
Added unit tests and tested to push data to a couchbase cluster though an 
azkaban job using hadoopJava and a snapshot of the library. Added log lines to 
verify expiration times are set correctly and tested pushing using cert based 
authentication. Please take a look at the following obfuscated data:
   
   `13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - 
INFO Setting ttl timestamp 1569234590 for document 'key1' and data 
{"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"pvFbXpmKT7ytUn8WYBic8A==","time":1566642590043}
   13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - 
INFO Setting ttl timestamp 1569013770 for document 'key2==' and data 
{"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"CtzzEq1rQ4qLXMT8zTD89w==","time":1566421770972}
   `
   and the records are on the bucket:
   `
   cbshell [ei-ltx1/lss-search-sales-seat-lead-activity]> get 
121084037038108585_91708167944609950_506_CtzzEq1rQ4qLXMT8zTD89w==
   {
 "action": "xxx",
 "memberId": xxx,
 "requestId": "CtzzEq1rQ4qLXMT8zTD89w==",
 "targetId": xxx,
 "time": 1566421770972
   }
   `
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…
   
   … cert based auth + enable TTL + dnsSrv
   
   JIRA ticket: https://issues.apache.org/jira/browse/GOBBLIN-880
   RB Changes:
   1 - Added logic to connect using certificate based auth to the buckets 
(Will need to bump  com.couchbase.client:java-client to a newer version like 
2.7.6) and associated configs
   2 - TTL  implementation
 * Added configs to allow setting a TTL (documentTTL) and also specify 
the timeunits (documentTTLUnits) of these settings
 * Added logic to specify the path to key to the field containing the 
source timestamp (documentTTLOriginField) and its units 
(documentTTLOriginUnits) to disambiguate between UNIX (sec) timestamps and 
other formats like timestamps in milliseconds.
   3 - Added missing dnsSrv config
   4 - Written proper documentation on 
gobblin-docs/writers/CouchbaseWriter.md
   5 - Brought in CouchbaseMock from Gradle and adapt existing unit tests.
   6 - Added getTimeUnit to ConfigUtils + Unit tests
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

2019-09-13 Thread Michael A Menarguez (Jira)

Michael A Menarguez created GOBBLIN-880:
---

 Summary: Bump CouchbaseWriter Couchbase SDK version + write docs + 
cert based auth + enable TTL + dnsSrv
 Key: GOBBLIN-880
 URL: https://issues.apache.org/jira/browse/GOBBLIN-880
 Project: Apache Gobblin
  Issue Type: Improvement
  Components: gobblin-couchbase
Reporter: Michael A Menarguez
Assignee: Shirshanka Das


h1. h1. CURRENT ISSUES

Currently CouchbaseWriter.java lacks the ability to do the following:
 # Use certificate based authentication
 # Set document expiry (TTL)
 ** based on write time
 ** based on an offset specified field contained in the record's data (JSON)
 ** (WILL NOT ADRESS) set expiry based on a field contained in the record's data
 # Set DNS SRV for bootstrap host discovery setting
 # Missing documentation on CouchbaseWriter usage
 # Testing does not bring in CouchbaseMock correctly and causes problems while 
bumping com.couchbase.client:java-client

h1. h1. PROPOSED SOLUTIONS
 # Add logic to connect using certificate based auth to the buckets (Will need 
to bump  com.couchbase.client:java-client to a newer version like 2.7.6) and 
associated configs
 # TTL implementation
 ## Add configs to allow setting a TTL (documentTTL) and also specify the 
timeunits (documentTTLUnits) of these settings
 ## Add logic to specify the path to key to the field containing the source 
timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to 
disambiguate between UNIX (sec) timestamps and other formats like timestamps in 
milliseconds.
 ## N/A but logic would be similar to (2)
 # Add missing dnsSrv config
 # Write proper documentation
 # Bring in CouchbaseMock from Gradle and adapt existing unit tests.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-09-13 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=312178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312178
 ]

ASF GitHub Bot logged work on GOBBLIN-875:
--

Author: ASF GitHub Bot
Created on: 13/Sep/19 16:53
Start Date: 13/Sep/19 16:53
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit 
container health metrics when running in cluster mode
URL: 
https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1)
 Report
   > Merging 
[#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc)
 will **increase** coverage by `0.09%`.
   > The diff coverage is `70.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2729  +/-   ##
   
   + Coverage 44.99%   45.08%   +0.09% 
   - Complexity 8742 8758  +16 
   
 Files  1884 1886   +2 
 Lines 7029570377  +82 
 Branches   7715 7718   +3 
   
   + Hits  3162931730 +101 
   + Misses3573535705  -30 
   - Partials   2931 2942  +11
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=)
 | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | |
   | 
[...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=)
 | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==)
 | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | |
   | 
[...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==)
 | `78.57% <78.57%>

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode

2019-09-13 Thread GitBox

codecov-io edited a comment on issue #2729: GOBBLIN-875: Emit container health 
metrics when running in cluster mode
URL: 
https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=h1)
 Report
   > Merging 
[#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr=desc)
 will **increase** coverage by `0.09%`.
   > The diff coverage is `70.42%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2729  +/-   ##
   
   + Coverage 44.99%   45.08%   +0.09% 
   - Complexity 8742 8758  +16 
   
 Files  1884 1886   +2 
 Lines 7029570377  +82 
 Branches   7715 7718   +3 
   
   + Hits  3162931730 +101 
   + Misses3573535705  -30 
   - Partials   2931 2942  +11
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=)
 | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | |
   | 
[...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=)
 | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==)
 | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | |
   | 
[...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==)
 | `78.57% <78.57%> (ø)` | `5 <5> (?)` | |
   | ... and [14 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? =

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

[GitHub] [incubator-gobblin] codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

[GitHub] [incubator-gobblin] jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses

[jira] [Created] (GOBBLIN-881) Add job tag field that can be used to filter job statuses

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] zxcware commented on issue #2733: Modify GaaS configuration so that it runs through the startup script

[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

[GitHub] [incubator-gobblin] codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…

[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition

[jira] [Commented] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

[GitHub] [incubator-gobblin] menarguez opened a new pull request #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…

[jira] [Created] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv

[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode

30 matches

Site Navigation

Mail list logo

Footer information