subject:"\[jira\] \[Work logged\] \(GOBBLIN\-1016\) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed"

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371966
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 23:13
Start Date: 14/Jan/20 23:13
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: 
https://github.com/apache/incubator-gobblin/pull/2863#issuecomment-572836083
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=h1)
 Report
   > Merging 
[#2863](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/4483268e3885adb63da7253df5ab0c00eb86aece?src=pr=desc)
 will **decrease** coverage by `41.61%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #2863   +/-   ##
   
   - Coverage 45.72%4.1%   -41.62% 
   + Complexity 9085 747 -8338 
   
 Files  19131917+4 
 Lines 72021   72188  +167 
 Branches   79437968   +25 
   
   - Hits  329292961-29968 
   - Misses36074   68908+32834 
   + Partials   3018 319 -2699
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=)
 | `0% <ø> (-54.17%)` | `0 <0> (-19)` | |
   | 
[...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=)
 | `0% <0%> (-46.16%)` | `0 <0> (-3)` | |
   | 
[.../gobblin/cluster/GobblinHelixMessagingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TWVzc2FnaW5nU2VydmljZS5qYXZh)
 | `0% <0%> (-65.17%)` | `0 <0> (-4)` | |
   | 
[...obblin/yarn/YarnAppSecurityManagerWithKeytabs.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkFwcFNlY3VyaXR5TWFuYWdlcldpdGhLZXl0YWJzLmphdmE=)
 | `0% <0%> (-20.59%)` | `0 <0> (-2)` | |
   | 
[...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh)
 | `0% <0%> (-19.95%)` | `0 <0> (-7)` | |
   | 
[...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==)
 | `0% <0%> (-100%)` | `0% <0%> (-2%)` | |
   | 
[...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==)
 | `0% <0%> (-100%)` | `0% <0%> (-5%)` | |
   | 
[...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh)
 | `0% <0%> (-100%)` | `0% <0%> (-3%)` | |
   | 
[...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=)
 | `0% <0%> (-100%)` | `0% <0%> (-1%)` | |
   |

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371883
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 21:26
Start Date: 14/Jan/20 21:26
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: 
https://github.com/apache/incubator-gobblin/pull/2863#issuecomment-572836083
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=h1)
 Report
   > Merging 
[#2863](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/4483268e3885adb63da7253df5ab0c00eb86aece?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `57.14%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2863  +/-   ##
   
   + Coverage 45.72%   45.77%   +0.05% 
   - Complexity 9085 9109  +24 
   
 Files  1913 1917   +4 
 Lines 7202172147 +126 
 Branches   7943 7963  +20 
   
   + Hits  3292933024  +95 
   - Misses3607436098  +24 
   - Partials   3018 3025   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=)
 | `54.16% <ø> (ø)` | `19 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/cluster/GobblinHelixMessagingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TWVzc2FnaW5nU2VydmljZS5qYXZh)
 | `63.04% <0%> (-2.13%)` | `4 <0> (ø)` | |
   | 
[...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=)
 | `48.75% <100%> (+2.59%)` | `6 <1> (+3)` | :arrow_up: |
   | 
[...obblin/yarn/YarnAppSecurityManagerWithKeytabs.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkFwcFNlY3VyaXR5TWFuYWdlcldpdGhLZXl0YWJzLmphdmE=)
 | `25.64% <33.33%> (+5.05%)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh)
 | `21.39% <90.9%> (+1.44%)` | `8 <0> (+1)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.2% <0%> (-0.68%)` | `24% <0%> (-1%)` | |
   | 
[...n/compaction/action/CompactionWatermarkAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25XYXRlcm1hcmtBY3Rpb24uamF2YQ==)
 | `77.04% <0%> (ø)` | `11% <0%> (?)` | |
   |

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371881
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 21:25
Start Date: 14/Jan/20 21:25
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: 
https://github.com/apache/incubator-gobblin/pull/2863#issuecomment-572836083
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=h1)
 Report
   > Merging 
[#2863](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/4483268e3885adb63da7253df5ab0c00eb86aece?src=pr=desc)
 will **decrease** coverage by `41.61%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #2863   +/-   ##
   
   - Coverage 45.72%4.1%   -41.62% 
   + Complexity 9085 747 -8338 
   
 Files  19131917+4 
 Lines 72021   72147  +126 
 Branches   79437963   +20 
   
   - Hits  329292961-29968 
   - Misses36074   68867+32793 
   + Partials   3018 319 -2699
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=)
 | `0% <ø> (-54.17%)` | `0 <0> (-19)` | |
   | 
[...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=)
 | `0% <0%> (-46.16%)` | `0 <0> (-3)` | |
   | 
[.../gobblin/cluster/GobblinHelixMessagingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TWVzc2FnaW5nU2VydmljZS5qYXZh)
 | `0% <0%> (-65.17%)` | `0 <0> (-4)` | |
   | 
[...obblin/yarn/YarnAppSecurityManagerWithKeytabs.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkFwcFNlY3VyaXR5TWFuYWdlcldpdGhLZXl0YWJzLmphdmE=)
 | `0% <0%> (-20.59%)` | `0 <0> (-2)` | |
   | 
[...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh)
 | `0% <0%> (-19.95%)` | `0 <0> (-7)` | |
   | 
[...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==)
 | `0% <0%> (-100%)` | `0% <0%> (-2%)` | |
   | 
[...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==)
 | `0% <0%> (-100%)` | `0% <0%> (-5%)` | |
   | 
[...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh)
 | `0% <0%> (-100%)` | `0% <0%> (-3%)` | |
   | 
[...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=)
 | `0% <0%> (-100%)` | `0% <0%> (-1%)` | |
   |

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371800=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371800
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 19:06
Start Date: 14/Jan/20 19:06
Worklog Time Spent: 10m 
  Work Description: htran1 commented on pull request #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: https://github.com/apache/incubator-gobblin/pull/2863#discussion_r366518825
 
 

 ##
 File path: 
gobblin-yarn/src/test/java/org/apache/gobblin/yarn/GobblinYarnAppLauncherTest.java
 ##
 @@ -265,17 +297,61 @@ public void testSendShutdownRequest() throws Exception {
 Assert.assertEquals(this.curatorFramework.checkExists()
 .forPath(String.format("/%s/CONTROLLER/MESSAGES", 
GobblinYarnAppLauncherTest.class.getSimpleName()))
 .getVersion(), 0);
-YarnSecurityManagerTest.GetControllerMessageNumFunc getCtrlMessageNum =
-new 
YarnSecurityManagerTest.GetControllerMessageNumFunc(GobblinYarnAppLauncherTest.class.getSimpleName(),
+YarnSecurityManagerTest.GetHelixMessageNumFunc getCtrlMessageNum =
+new 
YarnSecurityManagerTest.GetHelixMessageNumFunc(GobblinYarnAppLauncherTest.class.getSimpleName(),
 InstanceType.CONTROLLER, "",
 this.curatorFramework);
 AssertWithBackoff assertWithBackoff =
 
AssertWithBackoff.create().logger(LoggerFactory.getLogger("testSendShutdownRequest")).timeoutMs(2);
 assertWithBackoff.assertEquals(getCtrlMessageNum, 1, "1 controller message 
queued");
 
 // Give Helix sometime to handle the message
 assertWithBackoff.assertEquals(getCtrlMessageNum, 0, "all controller 
messages processed");
+
+this.helixManagerManagedHelix.connect();
+
this.helixManagerManagedHelix.getMessagingService().registerMessageHandlerFactory(GobblinHelixConstants.SHUTDOWN_MESSAGE_TYPE,
+new TestShutdownMessageHandlerFactory(this));
+
+this.gobblinYarnAppLauncherManagedHelix.connectHelixManager();
+this.gobblinYarnAppLauncherManagedHelix.sendShutdownRequest();
+
+Assert.assertEquals(this.curatorFramework.checkExists()
+.forPath(String.format("/%s/INSTANCES/%s/MESSAGES", 
this.configManagedHelix.getString(GobblinClusterConfigurationKeys.HELIX_CLUSTER_NAME_KEY),
 TEST_HELIX_INSTANCE_NAME_MANAGED))
+.getVersion(), 0);
+YarnSecurityManagerTest.GetHelixMessageNumFunc getInstanceMessageNum =
+new 
YarnSecurityManagerTest.GetHelixMessageNumFunc(this.configManagedHelix.getString(
+GobblinClusterConfigurationKeys.HELIX_CLUSTER_NAME_KEY),
+InstanceType.PARTICIPANT, TEST_HELIX_INSTANCE_NAME_MANAGED, 
this.curatorFramework);
+assertWithBackoff =
+
AssertWithBackoff.create().logger(LoggerFactory.getLogger("testSendShutdownRequest")).timeoutMs(2);
+assertWithBackoff.assertEquals(getInstanceMessageNum, 1, "1 controller 
message queued");
+
+// Give Helix sometime to handle the message
+assertWithBackoff.assertEquals(getInstanceMessageNum, 0, "all controller 
messages processed");
   }
 
+  /*static class GetInstanceMessageFunc implements Function {
 
 Review comment:
   Remove this block?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 371800)
Time Spent: 50m  (was: 40m)

> Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode 
> when Helix cluster is managed
> 
>
> Key: GOBBLIN-1016
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1016
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
> cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
> for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
> shutdown and token file update messages, the AppMaster should connect to the 
> Helix cluster as a PARTICIPANT, these message types need to be destined to 
> PARTICIPANT instead of CONTROLLER as is the case today.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371799
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 19:06
Start Date: 14/Jan/20 19:06
Worklog Time Spent: 10m 
  Work Description: htran1 commented on pull request #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: https://github.com/apache/incubator-gobblin/pull/2863#discussion_r366518174
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManagerWithKeytabs.java
 ##
 @@ -85,8 +91,11 @@ protected synchronized void renewDelegationToken() throws 
IOException, Interrupt
 
 if (!this.firstLogin) {
   // Send a message to the controller and all the participants if this is 
not the first login
 
 Review comment:
   Update comment since no longer sending to all the participants.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 371799)
Time Spent: 40m  (was: 0.5h)

> Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode 
> when Helix cluster is managed
> 
>
> Key: GOBBLIN-1016
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1016
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
> cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
> for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
> shutdown and token file update messages, the AppMaster should connect to the 
> Helix cluster as a PARTICIPANT, these message types need to be destined to 
> PARTICIPANT instead of CONTROLLER as is the case today.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371801
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 19:06
Start Date: 14/Jan/20 19:06
Worklog Time Spent: 10m 
  Work Description: htran1 commented on pull request #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: https://github.com/apache/incubator-gobblin/pull/2863#discussion_r366518122
 
 

 ##
 File path: 
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManagerWithKeytabs.java
 ##
 @@ -130,8 +139,11 @@ protected void login() throws IOException {
 
 if (!this.firstLogin) {
   // Send a message to the controller and all the participants
 
 Review comment:
   Update comment?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 371801)
Time Spent: 50m  (was: 40m)

> Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode 
> when Helix cluster is managed
> 
>
> Key: GOBBLIN-1016
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1016
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
> cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
> for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
> shutdown and token file update messages, the AppMaster should connect to the 
> Helix cluster as a PARTICIPANT, these message types need to be destined to 
> PARTICIPANT instead of CONTROLLER as is the case today.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-14 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=371798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371798
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 14/Jan/20 19:06
Start Date: 14/Jan/20 19:06
Worklog Time Spent: 10m 
  Work Description: htran1 commented on pull request #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: https://github.com/apache/incubator-gobblin/pull/2863#discussion_r366516926
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixMessagingService.java
 ##
 @@ -173,13 +173,19 @@ public GobblinHelixMessagingService(HelixManager 
manager) {
  * @param row row of currently persisted data
  * @return true if it matches, false otherwise
  */
+
 private boolean rowMatches(Criteria criteria, ZNRecordRow row) {
   String instanceName = normalizePattern(criteria.getInstanceName());
   String resourceName = normalizePattern(criteria.getResource());
   String partitionName = normalizePattern(criteria.getPartition());
   String partitionState = normalizePattern(criteria.getPartitionState());
-  return stringMatches(instanceName, row.getMapSubKey()) && 
stringMatches(resourceName, row.getRecordId())
-  && stringMatches(partitionName, row.getMapKey()) && 
stringMatches(partitionState, row.getMapValue());
+  return (stringMatches(instanceName, 
Strings.nullToEmpty(row.getMapSubKey())) ||
+  stringMatches(instanceName, Strings.nullToEmpty(row.getRecordId(
+  && stringMatches(resourceName, 
Strings.nullToEmpty(row.getRecordId()))
+  && stringMatches(partitionName, Strings.nullToEmpty(row.getMapKey()))
+  && stringMatches(partitionState, 
Strings.nullToEmpty(row.getMapValue()));
+  //return stringMatches(instanceName, row.getMapSubKey()) && 
stringMatches(resourceName, row.getRecordId())
 
 Review comment:
   Remove the commented out lines.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 371798)
Time Spent: 0.5h  (was: 20m)

> Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode 
> when Helix cluster is managed
> 
>
> Key: GOBBLIN-1016
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1016
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
> cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
> for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
> shutdown and token file update messages, the AppMaster should connect to the 
> Helix cluster as a PARTICIPANT, these message types need to be destined to 
> PARTICIPANT instead of CONTROLLER as is the case today.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=369536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369536
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 10/Jan/20 01:45
Start Date: 10/Jan/20 01:45
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: 
https://github.com/apache/incubator-gobblin/pull/2863#issuecomment-572836083
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=h1)
 Report
   > Merging 
[#2863](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/4483268e3885adb63da7253df5ab0c00eb86aece?src=pr=desc)
 will **increase** coverage by `0.04%`.
   > The diff coverage is `57.14%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2863  +/-   ##
   
   + Coverage 45.72%   45.76%   +0.04% 
   - Complexity 9085 9107  +22 
   
 Files  1913 1917   +4 
 Lines 7202172147 +126 
 Branches   7943 7963  +20 
   
   + Hits  3292933019  +90 
   - Misses3607436102  +28 
   - Partials   3018 3026   +8
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2863?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=)
 | `54.16% <ø> (ø)` | `19 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/cluster/GobblinHelixMessagingService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TWVzc2FnaW5nU2VydmljZS5qYXZh)
 | `63.04% <0%> (-2.13%)` | `4 <0> (ø)` | |
   | 
[...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=)
 | `48.75% <100%> (+2.59%)` | `6 <1> (+3)` | :arrow_up: |
   | 
[...obblin/yarn/YarnAppSecurityManagerWithKeytabs.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFybkFwcFNlY3VyaXR5TWFuYWdlcldpdGhLZXl0YWJzLmphdmE=)
 | `25.64% <33.33%> (+5.05%)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh)
 | `21.39% <90.9%> (+1.44%)` | `8 <0> (+1)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.2% <0%> (-0.68%)` | `24% <0%> (-1%)` | |
   | 
[...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2863/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==)
 | `55.85% <0%> (-0.46%)` | `26% <0%> (-1%)` | |
   |

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1016?focusedWorklogId=369510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369510
 ]

ASF GitHub Bot logged work on GOBBLIN-1016:
---

Author: ASF GitHub Bot
Created on: 10/Jan/20 01:14
Start Date: 10/Jan/20 01:14
Worklog Time Spent: 10m 
  Work Description: sv2000 commented on pull request #2863: GOBBLIN-1016: 
Allow Gobblin Application Master to join Helix cluster …
URL: https://github.com/apache/incubator-gobblin/pull/2863
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-1016
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
shutdown and token file update messages, the AppMaster should connect to the 
Helix cluster as a PARTICIPANT, these message types need to be destined to 
PARTICIPANT instead of CONTROLLER as is the case today.
   
   
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Added unit tests to YarnAppSecurityManagerTest and 
GobblinYarnAppLauncherTest classes.
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 369510)
Remaining Estimate: 0h
Time Spent: 10m

> Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode 
> when Helix cluster is managed
> 
>
> Key: GOBBLIN-1016
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1016
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In the managed cluster mode, the Gobblin Yarn Application Master joins Helix 
> cluster in the ADMINISTRATOR role. However, Helix does not support messaging 
> for instances which are not CONTROLLER or PARTICIPANT. To properly handle 
> shutdown and token file update messages, the AppMaster should connect to the 
> Helix cluster as a PARTICIPANT, these message types need to be destined to 
> PARTICIPANT instead of CONTROLLER as is the case today.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

[jira] [Work logged] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

9 matches

Site Navigation

Mail list logo

Footer information