[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode
codecov-io edited a comment on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=h1) Report > Merging [#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr&el=desc) will **increase** coverage by `0.09%`. > The diff coverage is `70.42%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2729 +/- ## + Coverage 44.99% 45.08% +0.09% - Complexity 8742 8758 +16 Files 1884 1886 +2 Lines 7029570377 +82 Branches 7715 7718 +3 + Hits 3162931730 +101 + Misses3573535705 -30 - Partials 2931 2942 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=) | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==) | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | | | [...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljc1NlcnZpY2UuamF2YQ==) | `78.57% <78.57%> (ø)` | `5 <5> (?)` | | | ... and [14 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-
[jira] [Work logged] (GOBBLIN-875) Emit container health metrics when running in cluster mode
[ https://issues.apache.org/jira/browse/GOBBLIN-875?focusedWorklogId=312178&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312178 ] ASF GitHub Bot logged work on GOBBLIN-875: -- Author: ASF GitHub Bot Created on: 13/Sep/19 16:53 Start Date: 13/Sep/19 16:53 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2729: GOBBLIN-875: Emit container health metrics when running in cluster mode URL: https://github.com/apache/incubator-gobblin/pull/2729#issuecomment-529597441 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=h1) Report > Merging [#2729](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a409908079df873ba0d5a465df3e682be0e0085d?src=pr&el=desc) will **increase** coverage by `0.09%`. > The diff coverage is `70.42%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2729 +/- ## + Coverage 44.99% 45.08% +0.09% - Complexity 8742 8758 +16 Files 1884 1886 +2 Lines 7029570377 +82 Branches 7715 7718 +3 + Hits 3162931730 +101 + Misses3573535705 -30 - Partials 2931 2942 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2729?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/cluster/GobblinClusterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJDb25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...che/gobblin/yarn/GobblinYarnConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/cluster/ContainerHealthMetrics.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGFpbmVySGVhbHRoTWV0cmljcy5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `53.91% <0%> (-0.51%)` | `27 <0> (ø)` | | | [...org/apache/gobblin/yarn/GobblinYarnTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5UYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/aws/GobblinAWSTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1hd3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYXdzL0dvYmJsaW5BV1NUYXNrUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/gobblin/yarn/GobblinApplicationMaster.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbkFwcGxpY2F0aW9uTWFzdGVyLmphdmE=) | `18.84% <100%> (+2.42%)` | `3 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/gobblin/yarn/YarnHelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkhlbGl4VXRpbHMuamF2YQ==) | `20% <100%> (+2.75%)` | `3 <1> (+1)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.81% <50%> (-0.45%)` | `28 <1> (-1)` | | | [...gobblin/cluster/ContainerHealthMetricsService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2729/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvQ29udGF
[jira] [Created] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv
Michael A Menarguez created GOBBLIN-880: --- Summary: Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv Key: GOBBLIN-880 URL: https://issues.apache.org/jira/browse/GOBBLIN-880 Project: Apache Gobblin Issue Type: Improvement Components: gobblin-couchbase Reporter: Michael A Menarguez Assignee: Shirshanka Das h1. h1. CURRENT ISSUES Currently CouchbaseWriter.java lacks the ability to do the following: # Use certificate based authentication # Set document expiry (TTL) ** based on write time ** based on an offset specified field contained in the record's data (JSON) ** (WILL NOT ADRESS) set expiry based on a field contained in the record's data # Set DNS SRV for bootstrap host discovery setting # Missing documentation on CouchbaseWriter usage # Testing does not bring in CouchbaseMock correctly and causes problems while bumping com.couchbase.client:java-client h1. h1. PROPOSED SOLUTIONS # Add logic to connect using certificate based auth to the buckets (Will need to bump com.couchbase.client:java-client to a newer version like 2.7.6) and associated configs # TTL implementation ## Add configs to allow setting a TTL (documentTTL) and also specify the timeunits (documentTTLUnits) of these settings ## Add logic to specify the path to key to the field containing the source timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to disambiguate between UNIX (sec) timestamps and other formats like timestamps in milliseconds. ## N/A but logic would be similar to (2) # Add missing dnsSrv config # Write proper documentation # Bring in CouchbaseMock from Gradle and adapt existing unit tests. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] menarguez opened a new pull request #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…
menarguez opened a new pull request #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… URL: https://github.com/apache/incubator-gobblin/pull/2734 ### JIRA - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-880 ### Description - [ ] Here are some details about my PR, including screenshots (if applicable): Plese read https://issues.apache.org/jira/browse/GOBBLIN-880 ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Added unit tests and tested to push data to a couchbase cluster though an azkaban job using hadoopJava and a snapshot of the library. Added log lines to verify expiration times are set correctly and tested pushing using cert based authentication. Please take a look at the following obfuscated data: `13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - INFO Setting ttl timestamp 1569234590 for document 'key1' and data {"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"pvFbXpmKT7ytUn8WYBic8A==","time":1566642590043} 13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - INFO Setting ttl timestamp 1569013770 for document 'key2==' and data {"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"CtzzEq1rQ4qLXMT8zTD89w==","time":1566421770972} ` and the records are on the bucket: ` cbshell [ei-ltx1/lss-search-sales-seat-lead-activity]> get 121084037038108585_91708167944609950_506_CtzzEq1rQ4qLXMT8zTD89w== { "action": "xxx", "memberId": xxx, "requestId": "CtzzEq1rQ4qLXMT8zTD89w==", "targetId": xxx, "time": 1566421770972 } ` ### Commits - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… … cert based auth + enable TTL + dnsSrv JIRA ticket: https://issues.apache.org/jira/browse/GOBBLIN-880 RB Changes: 1 - Added logic to connect using certificate based auth to the buckets (Will need to bump com.couchbase.client:java-client to a newer version like 2.7.6) and associated configs 2 - TTL implementation * Added configs to allow setting a TTL (documentTTL) and also specify the timeunits (documentTTLUnits) of these settings * Added logic to specify the path to key to the field containing the source timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to disambiguate between UNIX (sec) timestamps and other formats like timestamps in milliseconds. 3 - Added missing dnsSrv config 4 - Written proper documentation on gobblin-docs/writers/CouchbaseWriter.md 5 - Brought in CouchbaseMock from Gradle and adapt existing unit tests. 6 - Added getTimeUnit to ConfigUtils + Unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv
[ https://issues.apache.org/jira/browse/GOBBLIN-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929538#comment-16929538 ] Michael A Menarguez commented on GOBBLIN-880: - Submited PR [https://github.com/apache/incubator-gobblin/pull/2734] > Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + > enable TTL + dnsSrv > --- > > Key: GOBBLIN-880 > URL: https://issues.apache.org/jira/browse/GOBBLIN-880 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-couchbase >Reporter: Michael A Menarguez >Assignee: Shirshanka Das >Priority: Major > Labels: Couchbase > Original Estimate: 168h > Time Spent: 10m > Remaining Estimate: 167h 50m > > h1. h1. CURRENT ISSUES > Currently CouchbaseWriter.java lacks the ability to do the following: > # Use certificate based authentication > # Set document expiry (TTL) > ** based on write time > ** based on an offset specified field contained in the record's data (JSON) > ** (WILL NOT ADRESS) set expiry based on a field contained in the record's > data > # Set DNS SRV for bootstrap host discovery setting > # Missing documentation on CouchbaseWriter usage > # Testing does not bring in CouchbaseMock correctly and causes problems > while bumping com.couchbase.client:java-client > h1. h1. PROPOSED SOLUTIONS > # Add logic to connect using certificate based auth to the buckets (Will > need to bump com.couchbase.client:java-client to a newer version like 2.7.6) > and associated configs > # TTL implementation > ## Add configs to allow setting a TTL (documentTTL) and also specify the > timeunits (documentTTLUnits) of these settings > ## Add logic to specify the path to key to the field containing the source > timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to > disambiguate between UNIX (sec) timestamps and other formats like timestamps > in milliseconds. > ## N/A but logic would be similar to (2) > # Add missing dnsSrv config > # Write proper documentation > # Bring in CouchbaseMock from Gradle and adapt existing unit tests. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv
[ https://issues.apache.org/jira/browse/GOBBLIN-880?focusedWorklogId=312348&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312348 ] ASF GitHub Bot logged work on GOBBLIN-880: -- Author: ASF GitHub Bot Created on: 13/Sep/19 21:14 Start Date: 13/Sep/19 21:14 Worklog Time Spent: 10m Work Description: menarguez commented on pull request #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… URL: https://github.com/apache/incubator-gobblin/pull/2734 ### JIRA - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-880 ### Description - [ ] Here are some details about my PR, including screenshots (if applicable): Plese read https://issues.apache.org/jira/browse/GOBBLIN-880 ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Added unit tests and tested to push data to a couchbase cluster though an azkaban job using hadoopJava and a snapshot of the library. Added log lines to verify expiration times are set correctly and tested pushing using cert based authentication. Please take a look at the following obfuscated data: `13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - INFO Setting ttl timestamp 1569234590 for document 'key1' and data {"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"pvFbXpmKT7ytUn8WYBic8A==","time":1566642590043} 13-09-2019 12:42:11 PDT lss-seat-lead-history_gobblin-couchbase-push INFO - INFO Setting ttl timestamp 1569013770 for document 'key2==' and data {"memberId":xxx,"targetId":xxx,"action":"xxx","requestId":"CtzzEq1rQ4qLXMT8zTD89w==","time":1566421770972} ` and the records are on the bucket: ` cbshell [ei-ltx1/lss-search-sales-seat-lead-activity]> get 121084037038108585_91708167944609950_506_CtzzEq1rQ4qLXMT8zTD89w== { "action": "xxx", "memberId": xxx, "requestId": "CtzzEq1rQ4qLXMT8zTD89w==", "targetId": xxx, "time": 1566421770972 } ` ### Commits - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… … cert based auth + enable TTL + dnsSrv JIRA ticket: https://issues.apache.org/jira/browse/GOBBLIN-880 RB Changes: 1 - Added logic to connect using certificate based auth to the buckets (Will need to bump com.couchbase.client:java-client to a newer version like 2.7.6) and associated configs 2 - TTL implementation * Added configs to allow setting a TTL (documentTTL) and also specify the timeunits (documentTTLUnits) of these settings * Added logic to specify the path to key to the field containing the source timestamp (documentTTLOriginField) and its units (documentTTLOriginUnits) to disambiguate between UNIX (sec) timestamps and other formats like timestamps in milliseconds. 3 - Added missing dnsSrv config 4 - Written proper documentation on gobblin-docs/writers/CouchbaseWriter.md 5 - Brought in CouchbaseMock from Gradle and adapt existing unit tests. 6 - Added getTimeUnit to ConfigUtils + Unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312348) Remaining Estimate: 167h 50m (was: 168h) Time Spent: 10m > Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + > enable TTL + dnsSrv > --- > > Key: GOBBLIN-880 > URL: https://issues.apache.org/jira/browse/GOBBLIN-880 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-couchbase >Reporter: Michael A Menarguez >Assignee: Shirshanka Das >
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=h1) Report > Merging [#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr&el=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `1.78%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2722 +/- ## - Coverage 45.04% 44.96% -0.08% - Complexity 8739 8753 +14 Files 1880 1884 +4 Lines 7020570454 +249 Branches 7707 7730 +23 + Hits 3162331680 +57 - Misses3565135831 +180 - Partials 2931 2943 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | | | [...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh) | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | | | [...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==) | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | ... and [20 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree-more)
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312359 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 21:44 Start Date: 13/Sep/19 21:44 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=h1) Report > Merging [#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr&el=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `1.78%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2722 +/- ## - Coverage 45.04% 44.96% -0.08% - Complexity 8739 8753 +14 Files 1880 1884 +4 Lines 7020570454 +249 Branches 7707 7730 +23 + Hits 3162331680 +57 - Misses3565135831 +180 - Partials 2931 2943 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | | | [...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh) | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | | | [...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==) | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://
[jira] [Work logged] (GOBBLIN-880) Bump CouchbaseWriter Couchbase SDK version + write docs + cert based auth + enable TTL + dnsSrv
[ https://issues.apache.org/jira/browse/GOBBLIN-880?focusedWorklogId=312362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312362 ] ASF GitHub Bot logged work on GOBBLIN-880: -- Author: ASF GitHub Bot Created on: 13/Sep/19 21:47 Start Date: 13/Sep/19 21:47 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… URL: https://github.com/apache/incubator-gobblin/pull/2734#issuecomment-531401555 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=h1) Report > Merging [#2734](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `59.4%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2734 +/- ## + Coverage 45.04% 45.07% +0.03% - Complexity 8751 8761 +10 Files 1884 1884 Lines 7031170379 +68 Branches 7716 7723 +7 + Hits 3167231726 +54 - Misses3569735708 +11 - Partials 2942 2945 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/couchbase/writer/CouchbaseWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQnVpbGRlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../couchbase/writer/CouchbaseEnvironmentFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlRW52aXJvbm1lbnRGYWN0b3J5LmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==) | `95.65% <100%> (+1.9%)` | `2 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `60.4% <100%> (+2.54%)` | `40 <3> (+3)` | :arrow_up: | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=) | `64.39% <61.97%> (-4.22%)` | `15 <4> (+4)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.72% <0%> (+1.4%)` | `28% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <
[GitHub] [incubator-gobblin] codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +…
codecov-io commented on issue #2734: GOBBLIN-880 Bump CouchbaseWriter Couchbase SDK version + write docs +… URL: https://github.com/apache/incubator-gobblin/pull/2734#issuecomment-531401555 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=h1) Report > Merging [#2734](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `59.4%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2734 +/- ## + Coverage 45.04% 45.07% +0.03% - Complexity 8751 8761 +10 Files 1884 1884 Lines 7031170379 +68 Branches 7716 7723 +7 + Hits 3167231726 +54 - Misses3569735708 +11 - Partials 2942 2945 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...bblin/couchbase/writer/CouchbaseWriterBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQnVpbGRlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../couchbase/writer/CouchbaseEnvironmentFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlRW52aXJvbm1lbnRGYWN0b3J5LmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==) | `95.65% <100%> (+1.9%)` | `2 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/util/ConfigUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQ29uZmlnVXRpbHMuamF2YQ==) | `60.4% <100%> (+2.54%)` | `40 <3> (+3)` | :arrow_up: | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=) | `64.39% <61.97%> (-4.22%)` | `15 <4> (+4)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.72% <0%> (+1.4%)` | `28% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2734/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+6.54%)` | `13% <0%> (+2%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2734?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2
[GitHub] [incubator-gobblin] zxcware commented on issue #2733: Modify GaaS configuration so that it runs through the startup script
zxcware commented on issue #2733: Modify GaaS configuration so that it runs through the startup script URL: https://github.com/apache/incubator-gobblin/pull/2733#issuecomment-531402997 Hi @sv2000 , please review and merge the pr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324384004 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); Review comment: This is where to comment the right use cases to enable `PK_CHUNKING`, as it's the entry point ``` // pk-chunking only supports start-time by source.querybased.start.value, and does not support end-time. // always ingest data later than or equal source.querybased.start.value. // we should only pk chunking based work units only in case of snapshot/full ingestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
zxcware commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324383638 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String column, long value, String v return dataTypeMap; } + + private String partitionPkChunkingJobId = null; + private Iterator partitionPkChunkingBatchIdResultIterator = null; + + private Iterator getRecordSetPkchunking(WorkUnit workUnit) throws RuntimeException { +if (partitionPkChunkingBatchIdResultIterator == null) { + partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID); + partitionPkChunkingBatchIdResultIterator = Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator(); +} +if (!partitionPkChunkingBatchIdResultIterator.hasNext()) { + return null; +} +try { + if (!bulkApiLogin()) { +throw new IllegalArgumentException("Invalid Login"); + } +} catch (Exception e) { + throw new RuntimeException(e); +} +String[] batchIdResultIdArray = partitionPkChunkingBatchIdResultIterator.next().split(":"); +String batchId = batchIdResultIdArray[0]; +String resultId = batchIdResultIdArray[1]; +List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit); +return rs.iterator(); + } + @Override public Iterator getRecordSetFromSourceApi(String schema, String entity, WorkUnit workUnit, List predicateList) throws IOException { log.debug("Getting salesforce data using bulk api"); -RecordSet rs = null; + +// new version of extractor: bulk api with pk-chunking in pre-partitioning of SalesforceSource +if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) { Review comment: Are you checking if `PK_CHUNKING_JOB_ID` exists in workUnit? If so, we can use `workUnit.contains(PK_CHUNKING_JOB_ID)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312399 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 22:09 Start Date: 13/Sep/19 22:09 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324384004 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); Review comment: This is where to comment the right use cases to enable `PK_CHUNKING`, as it's the entry point ``` // pk-chunking only supports start-time by source.querybased.start.value, and does not support end-time. // always ingest data later than or equal source.querybased.start.value. // we should only pk chunking based work units only in case of snapshot/full ingestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312399) Time Spent: 4h 10m (was: 4h) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 4h 10m > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312398 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 22:09 Start Date: 13/Sep/19 22:09 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324383638 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String column, long value, String v return dataTypeMap; } + + private String partitionPkChunkingJobId = null; + private Iterator partitionPkChunkingBatchIdResultIterator = null; + + private Iterator getRecordSetPkchunking(WorkUnit workUnit) throws RuntimeException { +if (partitionPkChunkingBatchIdResultIterator == null) { + partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID); + partitionPkChunkingBatchIdResultIterator = Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator(); +} +if (!partitionPkChunkingBatchIdResultIterator.hasNext()) { + return null; +} +try { + if (!bulkApiLogin()) { +throw new IllegalArgumentException("Invalid Login"); + } +} catch (Exception e) { + throw new RuntimeException(e); +} +String[] batchIdResultIdArray = partitionPkChunkingBatchIdResultIterator.next().split(":"); +String batchId = batchIdResultIdArray[0]; +String resultId = batchIdResultIdArray[1]; +List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit); +return rs.iterator(); + } + @Override public Iterator getRecordSetFromSourceApi(String schema, String entity, WorkUnit workUnit, List predicateList) throws IOException { log.debug("Getting salesforce data using bulk api"); -RecordSet rs = null; + +// new version of extractor: bulk api with pk-chunking in pre-partitioning of SalesforceSource +if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) { Review comment: Are you checking if `PK_CHUNKING_JOB_ID` exists in workUnit? If so, we can use `workUnit.contains(PK_CHUNKING_JOB_ID)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312398) Time Spent: 4h 10m (was: 4h) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 4h 10m > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385070 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String column, long value, String v return dataTypeMap; } + + private String partitionPkChunkingJobId = null; + private Iterator partitionPkChunkingBatchIdResultIterator = null; + + private Iterator getRecordSetPkchunking(WorkUnit workUnit) throws RuntimeException { +if (partitionPkChunkingBatchIdResultIterator == null) { + partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID); + partitionPkChunkingBatchIdResultIterator = Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator(); +} +if (!partitionPkChunkingBatchIdResultIterator.hasNext()) { + return null; +} +try { + if (!bulkApiLogin()) { +throw new IllegalArgumentException("Invalid Login"); + } +} catch (Exception e) { + throw new RuntimeException(e); +} +String[] batchIdResultIdArray = partitionPkChunkingBatchIdResultIterator.next().split(":"); +String batchId = batchIdResultIdArray[0]; +String resultId = batchIdResultIdArray[1]; +List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit); +return rs.iterator(); + } + @Override public Iterator getRecordSetFromSourceApi(String schema, String entity, WorkUnit workUnit, List predicateList) throws IOException { log.debug("Getting salesforce data using bulk api"); -RecordSet rs = null; + +// new version of extractor: bulk api with pk-chunking in pre-partitioning of SalesforceSource +if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) { Review comment: Thanks! will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312400 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 22:13 Start Date: 13/Sep/19 22:13 Worklog Time Spent: 10m Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385070 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceExtractor.java ## @@ -588,11 +576,41 @@ public String getTimestampPredicateCondition(String column, long value, String v return dataTypeMap; } + + private String partitionPkChunkingJobId = null; + private Iterator partitionPkChunkingBatchIdResultIterator = null; + + private Iterator getRecordSetPkchunking(WorkUnit workUnit) throws RuntimeException { +if (partitionPkChunkingBatchIdResultIterator == null) { + partitionPkChunkingJobId = workUnit.getProp(PK_CHUNKING_JOB_ID); + partitionPkChunkingBatchIdResultIterator = Arrays.stream(workUnit.getProp(PK_CHUNKING_BATCH_RESULT_IDS).split(",")).iterator(); +} +if (!partitionPkChunkingBatchIdResultIterator.hasNext()) { + return null; +} +try { + if (!bulkApiLogin()) { +throw new IllegalArgumentException("Invalid Login"); + } +} catch (Exception e) { + throw new RuntimeException(e); +} +String[] batchIdResultIdArray = partitionPkChunkingBatchIdResultIterator.next().split(":"); +String batchId = batchIdResultIdArray[0]; +String resultId = batchIdResultIdArray[1]; +List rs = fetchPkChunkingResultSetWithRetry(bulkConnection, partitionPkChunkingJobId, batchId, resultId, fetchRetryLimit); +return rs.iterator(); + } + @Override public Iterator getRecordSetFromSourceApi(String schema, String entity, WorkUnit workUnit, List predicateList) throws IOException { log.debug("Getting salesforce data using bulk api"); -RecordSet rs = null; + +// new version of extractor: bulk api with pk-chunking in pre-partitioning of SalesforceSource +if (!workUnit.getProp(PK_CHUNKING_JOB_ID, "").equals("")) { Review comment: Thanks! will do. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312400) Time Spent: 4h 20m (was: 4h 10m) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 4h 20m > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385831 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); Review comment: Thanks, will add it! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312402&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312402 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 22:17 Start Date: 13/Sep/19 22:17 Worklog Time Spent: 10m Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324385831 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,98 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); Review comment: Thanks, will add it! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312402) Time Spent: 4.5h (was: 4h 20m) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 4.5h > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
codecov-io edited a comment on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=h1) Report > Merging [#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr&el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `1.78%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2722 +/- ## - Coverage 45.04% 44.96% -0.09% - Complexity 8739 8752 +13 Files 1880 1884 +4 Lines 7020570454 +249 Branches 7707 7730 +23 + Hits 3162331678 +55 - Misses3565135831 +180 - Partials 2931 2945 +14 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | | | [...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh) | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | | | [...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==) | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree-more)
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312419&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312419 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 22:53 Start Date: 13/Sep/19 22:53 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#issuecomment-531069100 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=h1) Report > Merging [#2722](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9bf9a882427e98e7f4ef089c4ca1bde42f4b36a3?src=pr&el=desc) will **decrease** coverage by `0.08%`. > The diff coverage is `1.78%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2722 +/- ## - Coverage 45.04% 44.96% -0.09% - Complexity 8739 8752 +13 Files 1880 1884 +4 Lines 7020570454 +249 Branches 7707 7730 +23 + Hits 3162331678 +55 - Misses3565135831 +180 - Partials 2931 2945 +14 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2722?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...obblin/salesforce/SalesforceConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `19.74% <5.66%> (-3.02%)` | `12 <1> (+1)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `82.14% <0%> (-7.15%)` | `11% <0%> (-1%)` | | | [...apache/gobblin/runtime/local/LocalJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxKb2JMYXVuY2hlci5qYXZh) | `61.81% <0%> (-2.34%)` | `5% <0%> (ø)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-0.72%)` | `15% <0%> (ø)` | | | [...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==) | `31.69% <0%> (-0.15%)` | `12% <0%> (ø)` | | | [...e/modules/flowgraph/datanodes/fs/AdlsDataNode.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0FkbHNEYXRhTm9kZS5qYXZh) | `50% <0%> (ø)` | `2% <0%> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2722/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://
[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition
arekusuri commented on a change in pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324393059 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,101 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + // pk-chunking only supports start-time by source.querybased.start.value, and does not support end-time. + // always ingest data later than or equal source.querybased.start.value. + // we should only pk chunking based work units only in case of snapshot/full ingestion + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); +} else { + return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark); +} + } + + /** + * generate workUnit with noQuery=true + */ + private List generateWorkUnitsPkChunking(SourceEntity sourceEntity, SourceState state, long previousWatermark) { + SalesforceBulkJobId salesforceBulkJobId = executeQueryWithPkChunking(state, previousWatermark); + List ret = createWorkUnits(sourceEntity, state, salesforceBulkJobId); + return ret; + } + + private SalesforceBulkJobId executeQueryWithPkChunking( + SourceState sourceState, + long previousWatermark + ) throws RuntimeException { +State state = new State(sourceState); +WorkUnit workUnit = WorkUnit.createEmpty(); +try { + WorkUnitState workUnitState = new WorkUnitState(workUnit, state); + workUnitState.setId("Execute pk-chunking"); Review comment: Hi @zxcware is this OK? I am trying to set id for workUnit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-865) Add feature that enables PK-chunking in partition
[ https://issues.apache.org/jira/browse/GOBBLIN-865?focusedWorklogId=312446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312446 ] ASF GitHub Bot logged work on GOBBLIN-865: -- Author: ASF GitHub Bot Created on: 13/Sep/19 23:45 Start Date: 13/Sep/19 23:45 Worklog Time Spent: 10m Work Description: arekusuri commented on pull request #2722: GOBBLIN-865: Add feature that enables PK-chunking in partition URL: https://github.com/apache/incubator-gobblin/pull/2722#discussion_r324393059 ## File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java ## @@ -146,12 +156,101 @@ protected void addLineageSourceInfo(SourceState sourceState, SourceEntity entity @Override protected List generateWorkUnits(SourceEntity sourceEntity, SourceState state, long previousWatermark) { +String partitionType = state.getProp(PARTITION_TYPE, ""); +if (partitionType.equals("PK_CHUNKING")) { + // pk-chunking only supports start-time by source.querybased.start.value, and does not support end-time. + // always ingest data later than or equal source.querybased.start.value. + // we should only pk chunking based work units only in case of snapshot/full ingestion + return generateWorkUnitsPkChunking(sourceEntity, state, previousWatermark); +} else { + return generateWorkUnitsStrategy(sourceEntity, state, previousWatermark); +} + } + + /** + * generate workUnit with noQuery=true + */ + private List generateWorkUnitsPkChunking(SourceEntity sourceEntity, SourceState state, long previousWatermark) { + SalesforceBulkJobId salesforceBulkJobId = executeQueryWithPkChunking(state, previousWatermark); + List ret = createWorkUnits(sourceEntity, state, salesforceBulkJobId); + return ret; + } + + private SalesforceBulkJobId executeQueryWithPkChunking( + SourceState sourceState, + long previousWatermark + ) throws RuntimeException { +State state = new State(sourceState); +WorkUnit workUnit = WorkUnit.createEmpty(); +try { + WorkUnitState workUnitState = new WorkUnitState(workUnit, state); + workUnitState.setId("Execute pk-chunking"); Review comment: Hi @zxcware is this OK? I am trying to set id for workUnit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312446) Time Spent: 4h 50m (was: 4h 40m) > Add feature that enables PK-chunking in partition > -- > > Key: GOBBLIN-865 > URL: https://issues.apache.org/jira/browse/GOBBLIN-865 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Li >Priority: Major > Labels: salesforce > Time Spent: 4h 50m > Remaining Estimate: 0h > > In SFDC(salesforce) connector, we have partitioning mechanisms to split a > giant query to multiple sub queries. There are 3 mechanisms: > * simple partition (equally split by time) > * dynamic pre-partition (generate histogram and split by row numbers) > * user specified partition (set up time range in job file) > However there are tables like Task and Contract are failing time to time to > fetch full data. > We may want to utilize PK-chunking to partition the query. > > The pk-chunking doc from SFDC - > [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (GOBBLIN-881) Add job tag field that can be used to filter job statuses
Jack Moseley created GOBBLIN-881: Summary: Add job tag field that can be used to filter job statuses Key: GOBBLIN-881 URL: https://issues.apache.org/jira/browse/GOBBLIN-881 Project: Apache Gobblin Issue Type: Bug Reporter: Jack Moseley -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses
jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531424495 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses
jack-moseley opened a new pull request #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [x] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-881 ### Description - [x] Here are some details about my PR, including screenshots (if applicable): With multi-dataset support in gaas, it is more convenient to have an extra field that can be used to filter when querying the job status (this would be dataset name in most cases). This PR allows jobs to include `job.tag` field then when querying jobStatus, specifying tag in the query will return only job statuses with that tag. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Updated test ### Commits - [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses
[ https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312450 ] ASF GitHub Bot logged work on GOBBLIN-881: -- Author: ASF GitHub Bot Created on: 13/Sep/19 23:53 Start Date: 13/Sep/19 23:53 Worklog Time Spent: 10m Work Description: jack-moseley commented on pull request #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [x] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-881 ### Description - [x] Here are some details about my PR, including screenshots (if applicable): With multi-dataset support in gaas, it is more convenient to have an extra field that can be used to filter when querying the job status (this would be dataset name in most cases). This PR allows jobs to include `job.tag` field then when querying jobStatus, specifying tag in the query will return only job statuses with that tag. ### Tests - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Updated test ### Commits - [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312450) Remaining Estimate: 0h Time Spent: 10m > Add job tag field that can be used to filter job statuses > - > > Key: GOBBLIN-881 > URL: https://issues.apache.org/jira/browse/GOBBLIN-881 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses
[ https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312451&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312451 ] ASF GitHub Bot logged work on GOBBLIN-881: -- Author: ASF GitHub Bot Created on: 13/Sep/19 23:53 Start Date: 13/Sep/19 23:53 Worklog Time Spent: 10m Work Description: jack-moseley commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531424495 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312451) Time Spent: 20m (was: 10m) > Add job tag field that can be used to filter job statuses > - > > Key: GOBBLIN-881 > URL: https://issues.apache.org/jira/browse/GOBBLIN-881 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [incubator-gobblin] codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses
codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531431862 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=h1) Report > Merging [#2735](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `30.76%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2735 +/- ## + Coverage 45.04% 45.05% +0.01% - Complexity 8751 8752 +1 Files 1884 1884 Lines 7031170318 +7 Branches 7716 7718 +2 + Hits 3167231685 +13 + Misses3569735692 -5 + Partials 2942 2941 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/event/TimingEvent.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9UaW1pbmdFdmVudC5qYXZh) | `70% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==) | `22.22% <0%> (-1.31%)` | `4 <0> (ø)` | | | [...ervice/modules/orchestration/TimingEventUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL1RpbWluZ0V2ZW50VXRpbHMuamF2YQ==) | `95.65% <100%> (+0.19%)` | `4 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=) | `72.34% <100%> (ø)` | `7 <0> (ø)` | :arrow_down: | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `74.19% <40%> (-7.95%)` | `11 <4> (ø)` | | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyLmphdmE=) | `66.27% <0%> (-2.33%)` | `11% <0%> (ø)` | | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | | | ... and [6 more](ht
[jira] [Work logged] (GOBBLIN-881) Add job tag field that can be used to filter job statuses
[ https://issues.apache.org/jira/browse/GOBBLIN-881?focusedWorklogId=312459&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312459 ] ASF GitHub Bot logged work on GOBBLIN-881: -- Author: ASF GitHub Bot Created on: 14/Sep/19 00:58 Start Date: 14/Sep/19 00:58 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2735: [GOBBLIN-881] Add job tag field that can be used to filter job statuses URL: https://github.com/apache/incubator-gobblin/pull/2735#issuecomment-531431862 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=h1) Report > Merging [#2735](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/c35871920cc5f8378a4f9962658b840994f071f5?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `30.76%`. [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2735 +/- ## + Coverage 45.04% 45.05% +0.01% - Complexity 8751 8752 +1 Files 1884 1884 Lines 7031170318 +7 Branches 7716 7718 +2 + Hits 3167231685 +13 + Misses3569735692 -5 + Partials 2942 2941 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2735?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/event/TimingEvent.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9UaW1pbmdFdmVudC5qYXZh) | `70% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2 <0> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==) | `22.22% <0%> (-1.31%)` | `4 <0> (ø)` | | | [...ervice/modules/orchestration/TimingEventUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL1RpbWluZ0V2ZW50VXRpbHMuamF2YQ==) | `95.65% <100%> (+0.19%)` | `4 <0> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=) | `72.34% <100%> (ø)` | `7 <0> (ø)` | :arrow_down: | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `74.19% <40%> (-7.95%)` | `11 <4> (ø)` | | | [...ache/gobblin/couchbase/writer/CouchbaseWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2735/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y