date:20191220

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1011?focusedWorklogId=362079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362079
 ]

ASF GitHub Bot logged work on GOBBLIN-1011:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 06:15
Start Date: 21/Dec/19 06:15
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2856: [GOBBLIN-1011] 
adjust compaction flow to work with virtual partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2856#issuecomment-568155535
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=h1)
 Report
   > Merging 
[#2856](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/070009231770e04aeb9a74a9d03865cb74cf4094?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `63.01%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2856  +/-   ##
   
   + Coverage 45.57%   45.63%   +0.05% 
   - Complexity 9032 9053  +21 
   
 Files  1908 1908  
 Lines 7172971776  +47 
 Branches   7912 7921   +9 
   
   + Hits  3269332757  +64 
   + Misses3603136007  -24 
   - Partials   3005 3012   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...n/compaction/suite/CompactionSuiteBaseFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZUZhY3RvcnkuamF2YQ==)
 | `100% <ø> (ø)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/compaction/suite/CompactionSuiteBase.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZS5qYXZh)
 | `100% <100%> (ø)` | `8 <2> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/runtime/task/TaskIFaceWrapper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGFzay9UYXNrSUZhY2VXcmFwcGVyLmphdmE=)
 | `68.62% <100%> (+1.96%)` | `14 <1> (+1)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `73.07% <62.5%> (+9.91%)` | `12 <5> (+8)` | :arrow_up: |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `35.58% <66.66%> (+3.88%)` | `15 <0> (+3)` | :arrow_up: |
   | 
[.../action/CompactionCompleteFileOperationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25Db21wbGV0ZUZpbGVPcGVyYXRpb25BY3Rpb24uamF2YQ==)
 | `62.5% <66.66%> (+0.2%)` | `5 <2> (+1)` | :arrow_up: |
   | 
[...mpaction/action/CompactionMarkDirectoryAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25NYXJrRGlyZWN0b3J5QWN0aW9uLmphdmE=)
 | `44.44% <66.66%> (+2.77%)` | `5 <2> (+1)` | :arrow_up: |
   | 
[...ction/action/CompactionHiveRegistrationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25IaXZlUmVnaXN0cmF0aW9uQWN0aW9uLmphdmE=)
 | `34.61% <66.66%> (+4.18%)` | `5 <2> (+1)` | :arrow_up: |

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

2019-12-20 Thread GitBox

codecov-io edited a comment on issue #2856: [GOBBLIN-1011] adjust compaction 
flow to work with virtual partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2856#issuecomment-568155535
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=h1)
 Report
   > Merging 
[#2856](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/070009231770e04aeb9a74a9d03865cb74cf4094?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `63.01%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2856  +/-   ##
   
   + Coverage 45.57%   45.63%   +0.05% 
   - Complexity 9032 9053  +21 
   
 Files  1908 1908  
 Lines 7172971776  +47 
 Branches   7912 7921   +9 
   
   + Hits  3269332757  +64 
   + Misses3603136007  -24 
   - Partials   3005 3012   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...n/compaction/suite/CompactionSuiteBaseFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZUZhY3RvcnkuamF2YQ==)
 | `100% <ø> (ø)` | `2 <0> (ø)` | :arrow_down: |
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/compaction/suite/CompactionSuiteBase.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZS5qYXZh)
 | `100% <100%> (ø)` | `8 <2> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/runtime/task/TaskIFaceWrapper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGFzay9UYXNrSUZhY2VXcmFwcGVyLmphdmE=)
 | `68.62% <100%> (+1.96%)` | `14 <1> (+1)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `73.07% <62.5%> (+9.91%)` | `12 <5> (+8)` | :arrow_up: |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `35.58% <66.66%> (+3.88%)` | `15 <0> (+3)` | :arrow_up: |
   | 
[.../action/CompactionCompleteFileOperationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25Db21wbGV0ZUZpbGVPcGVyYXRpb25BY3Rpb24uamF2YQ==)
 | `62.5% <66.66%> (+0.2%)` | `5 <2> (+1)` | :arrow_up: |
   | 
[...mpaction/action/CompactionMarkDirectoryAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25NYXJrRGlyZWN0b3J5QWN0aW9uLmphdmE=)
 | `44.44% <66.66%> (+2.77%)` | `5 <2> (+1)` | :arrow_up: |
   | 
[...ction/action/CompactionHiveRegistrationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25IaXZlUmVnaXN0cmF0aW9uQWN0aW9uLmphdmE=)
 | `34.61% <66.66%> (+4.18%)` | `5 <2> (+1)` | :arrow_up: |
   | 
[...c/main/java/org/apache/gobblin/hive/HiveTable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVUYWJsZS5qYXZh)
 | `48.88% <71.42%> (+4.15%)` | `8 <1> (+1)` | :arrow_up: |
   | ... and [15 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree-more)
 | |

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1011?focusedWorklogId=362077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362077
 ]

ASF GitHub Bot logged work on GOBBLIN-1011:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 06:00
Start Date: 21/Dec/19 06:00
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2856: [GOBBLIN-1011] 
adjust compaction flow to work with virtual partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2856#issuecomment-568155535
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=h1)
 Report
   > Merging 
[#2856](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/070009231770e04aeb9a74a9d03865cb74cf4094?src=pr=desc)
 will **decrease** coverage by `41.45%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #2856   +/-   ##
   
   - Coverage 45.57%   4.12%   -41.46% 
   + Complexity 9032 747 -8285 
   
 Files  19081908   
 Lines 71729   71776   +47 
 Branches   79127921+9 
   
   - Hits  326932961-29732 
   - Misses36031   68496+32465 
   + Partials   3005 319 -2686
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...n/compaction/suite/CompactionSuiteBaseFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZUZhY3RvcnkuamF2YQ==)
 | `0% <ø> (-100%)` | `0 <0> (-2)` | |
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/compaction/suite/CompactionSuiteBase.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZS5qYXZh)
 | `0% <0%> (-100%)` | `0 <0> (-8)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `0% <0%> (-31.7%)` | `0 <0> (-12)` | |
   | 
[...ompaction/verify/CompactionAuditCountVerifier.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vdmVyaWZ5L0NvbXBhY3Rpb25BdWRpdENvdW50VmVyaWZpZXIuamF2YQ==)
 | `0% <0%> (-80%)` | `0 <0> (-13)` | |
   | 
[.../action/CompactionCompleteFileOperationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25Db21wbGV0ZUZpbGVPcGVyYXRpb25BY3Rpb24uamF2YQ==)
 | `0% <0%> (-62.3%)` | `0 <0> (-4)` | |
   | 
[.../apache/gobblin/runtime/task/TaskIFaceWrapper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGFzay9UYXNrSUZhY2VXcmFwcGVyLmphdmE=)
 | `0% <0%> (-66.67%)` | `0 <0> (-13)` | |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `0% <0%> (-63.16%)` | `0 <0> (-4)` | |
   | 
[...mpaction/action/CompactionMarkDirectoryAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25NYXJrRGlyZWN0b3J5QWN0aW9uLmphdmE=)
 | `0% <0%> (-41.67%)` | `0 <0> (-4)` | |
   |

[GitHub] [incubator-gobblin] codecov-io commented on issue #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

2019-12-20 Thread GitBox

codecov-io commented on issue #2856: [GOBBLIN-1011] adjust compaction flow to 
work with virtual partition
URL: 
https://github.com/apache/incubator-gobblin/pull/2856#issuecomment-568155535
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=h1)
 Report
   > Merging 
[#2856](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/070009231770e04aeb9a74a9d03865cb74cf4094?src=pr=desc)
 will **decrease** coverage by `41.45%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #2856   +/-   ##
   
   - Coverage 45.57%   4.12%   -41.46% 
   + Complexity 9032 747 -8285 
   
 Files  19081908   
 Lines 71729   71776   +47 
 Branches   79127921+9 
   
   - Hits  326932961-29732 
   - Misses36031   68496+32465 
   + Partials   3005 319 -2686
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...n/compaction/suite/CompactionSuiteBaseFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZUZhY3RvcnkuamF2YQ==)
 | `0% <ø> (-100%)` | `0 <0> (-2)` | |
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../gobblin/compaction/suite/CompactionSuiteBase.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlQmFzZS5qYXZh)
 | `0% <0%> (-100%)` | `0 <0> (-8)` | |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `0% <0%> (-31.7%)` | `0 <0> (-12)` | |
   | 
[...ompaction/verify/CompactionAuditCountVerifier.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vdmVyaWZ5L0NvbXBhY3Rpb25BdWRpdENvdW50VmVyaWZpZXIuamF2YQ==)
 | `0% <0%> (-80%)` | `0 <0> (-13)` | |
   | 
[.../action/CompactionCompleteFileOperationAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25Db21wbGV0ZUZpbGVPcGVyYXRpb25BY3Rpb24uamF2YQ==)
 | `0% <0%> (-62.3%)` | `0 <0> (-4)` | |
   | 
[.../apache/gobblin/runtime/task/TaskIFaceWrapper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGFzay9UYXNrSUZhY2VXcmFwcGVyLmphdmE=)
 | `0% <0%> (-66.67%)` | `0 <0> (-13)` | |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `0% <0%> (-63.16%)` | `0 <0> (-4)` | |
   | 
[...mpaction/action/CompactionMarkDirectoryAction.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vYWN0aW9uL0NvbXBhY3Rpb25NYXJrRGlyZWN0b3J5QWN0aW9uLmphdmE=)
 | `0% <0%> (-41.67%)` | `0 <0> (-4)` | |
   | 
[...c/main/java/org/apache/gobblin/hive/HiveTable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVUYWJsZS5qYXZh)
 | `0% <0%> (-44.74%)` | `0 <0> (-7)` | |
   | ... and [1118 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2856/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2856?src=pr=continue).
   > **Legend** - [Click

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1011?focusedWorklogId=362003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-362003
 ]

ASF GitHub Bot logged work on GOBBLIN-1011:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 01:34
Start Date: 21/Dec/19 01:34
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2856: [GOBBLIN-1011] 
adjust compaction flow to work with virtual partition
URL: https://github.com/apache/incubator-gobblin/pull/2856
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-1011
   
   
   ### Description
   - [x] Here are some details about my PR:
 - Update existing `CompactionVerifier`s and `CompactionCompleteAction`s to 
work with virtual simple file system dataset proply
 - Improve ser/de of `FileSystemDataset` in `CompactionSuiteBase`
 - Update gobblin-hive-registration to work with table parameters properly
   
   
   
   ### Tests
   - [x] My PR adds the following unit tests:
 - `TimeIteratorTest` covers functions in `TimeIterator`
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 362003)
Remaining Estimate: 0h
Time Spent: 10m

> Adjust compaction flow to work with virtual partition
> -
>
> Key: GOBBLIN-1011
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1011
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> - Update existing `CompactionVerifier`s and `CompactionCompleteAction`s to 
> work with virtual simple file system dataset proply
> - Improve ser/de of `FileSystemDataset` in `CompactionSuiteBase`
> - Update gobblin-hive-registration to work with table parameters properly



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] zxcware opened a new pull request #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

2019-12-20 Thread GitBox

zxcware opened a new pull request #2856: [GOBBLIN-1011] adjust compaction flow 
to work with virtual partition
URL: https://github.com/apache/incubator-gobblin/pull/2856
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-1011
   
   
   ### Description
   - [x] Here are some details about my PR:
 - Update existing `CompactionVerifier`s and `CompactionCompleteAction`s to 
work with virtual simple file system dataset proply
 - Improve ser/de of `FileSystemDataset` in `CompactionSuiteBase`
 - Update gobblin-hive-registration to work with table parameters properly
   
   
   
   ### Tests
   - [x] My PR adds the following unit tests:
 - `TimeIteratorTest` covers functions in `TimeIterator`
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

2019-12-20 Thread Zhixiong Chen (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhixiong Chen updated GOBBLIN-1011:
---
Description: 
- Update existing `CompactionVerifier`s and `CompactionCompleteAction`s to work 
with virtual simple file system dataset proply
- Improve ser/de of `FileSystemDataset` in `CompactionSuiteBase`
- Update gobblin-hive-registration to work with table parameters properly



> Adjust compaction flow to work with virtual partition
> -
>
> Key: GOBBLIN-1011
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1011
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
>
> - Update existing `CompactionVerifier`s and `CompactionCompleteAction`s to 
> work with virtual simple file system dataset proply
> - Improve ser/de of `FileSystemDataset` in `CompactionSuiteBase`
> - Update gobblin-hive-registration to work with table parameters properly



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

2019-12-20 Thread Zhixiong Chen (Jira)

Zhixiong Chen created GOBBLIN-1011:
--

 Summary: Adjust compaction flow to work with virtual partition
 Key: GOBBLIN-1011
 URL: https://issues.apache.org/jira/browse/GOBBLIN-1011
 Project: Apache Gobblin
  Issue Type: Task
Reporter: Zhixiong Chen
Assignee: Zhixiong Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (GOBBLIN-1001) Implement TimePartitionGlobFinder

2019-12-20 Thread Hung Tran (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hung Tran resolved GOBBLIN-1001.

Fix Version/s: 0.15.0
   Resolution: Fixed

Issue resolved by pull request #2846
[https://github.com/apache/incubator-gobblin/pull/2846]

> Implement TimePartitionGlobFinder
> -
>
> Key: GOBBLIN-1001
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1001
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1001?focusedWorklogId=361998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361998
 ]

ASF GitHub Bot logged work on GOBBLIN-1001:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 01:21
Start Date: 21/Dec/19 01:21
Worklog Time Spent: 10m 
  Work Description: asfgit commented on pull request #2846: [GOBBLIN-1001] 
Implement TimePartitionGlobFinder
URL: https://github.com/apache/incubator-gobblin/pull/2846
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361998)
Time Spent: 3h 20m  (was: 3h 10m)

> Implement TimePartitionGlobFinder
> -
>
> Key: GOBBLIN-1001
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1001
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] asfgit closed pull request #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

2019-12-20 Thread GitBox

asfgit closed pull request #2846: [GOBBLIN-1001] Implement 
TimePartitionGlobFinder
URL: https://github.com/apache/incubator-gobblin/pull/2846
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1009?focusedWorklogId=361983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361983
 ]

ASF GitHub Bot logged work on GOBBLIN-1009:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 00:37
Start Date: 21/Dec/19 00:37
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2854: [GOBBLIN-1009] 
Create load balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360614396
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/ForwardRequestServlet.java
 ##
 @@ -0,0 +1,174 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Enumeration;
+import java.util.concurrent.Future;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.ServletException;
+import org.apache.commons.io.IOUtils;
+import org.apache.gobblin.util.HashingUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
+import org.apache.http.impl.nio.client.HttpAsyncClients;
+import org.apache.http.protocol.HTTP;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ForwardRequestServlet extends HttpServlet {
+
+  public CloseableHttpAsyncClient client;
+
+  private final String STATEFULSET_URL_PREFIX = "http://gaas-;;
+  // TODO: split this up into configuration or something
+  private final String STATEFULSET_URL_SUFFIX = 
".gaas.default.svc.cluster.local:6956";
+  private final String LOADBALANCER_PREFIX = "gobblinServiceLoadbalancer.";
+  private final String NUM_SCHEDULERS_KEY = "numSchedulers";
+  private final String FLOW_NAME_KEY = "flowName";
+  private final String FLOW_GROUP_KEY = "flowGroup";
+  private int numSchedulers;
+  protected final Logger _log;
+
+  ForwardRequestServlet(Config config, Optional log) {
+_log = log.isPresent() ? log.get() : LoggerFactory.getLogger(getClass());
+this.numSchedulers = config.getInt(LOADBALANCER_PREFIX + 
NUM_SCHEDULERS_KEY);
+_log.info("Load balancer started with {} servers", this.numSchedulers);
+this.client = HttpAsyncClients.createDefault();
+this.client.start();
+  }
+
+  protected void doGet(HttpServletRequest request, HttpServletResponse 
response) {
 
 Review comment:
   Is this an `override`? If so, let's add `@override` annotation for better 
code readability, avoiding future readers asking this question. This one 
applies to the other methods
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361983)
Remaining Estimate: 0h
Time Spent: 10m

> Create loadbalancer and shard FlowSpec schedules
> 
>
> Key: GOBBLIN-1009
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1009
> Project: Apache Gobblin
>  Issue Type: New Feature
>Reporter: William Lo
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Creates a load balancer to distribute incoming requests among multiple GaaS 
> instances.
> This allows there to be more than a single instance of GaaS running on a 
> kubernetes cluster without Apache Helix



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1009?focusedWorklogId=361982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361982
 ]

ASF GitHub Bot logged work on GOBBLIN-1009:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 00:37
Start Date: 21/Dec/19 00:37
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2854: [GOBBLIN-1009] 
Create load balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360615747
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/ForwardRequestServlet.java
 ##
 @@ -0,0 +1,174 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Enumeration;
+import java.util.concurrent.Future;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.ServletException;
+import org.apache.commons.io.IOUtils;
+import org.apache.gobblin.util.HashingUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
+import org.apache.http.impl.nio.client.HttpAsyncClients;
+import org.apache.http.protocol.HTTP;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ForwardRequestServlet extends HttpServlet {
+
+  public CloseableHttpAsyncClient client;
+
+  private final String STATEFULSET_URL_PREFIX = "http://gaas-;;
+  // TODO: split this up into configuration or something
+  private final String STATEFULSET_URL_SUFFIX = 
".gaas.default.svc.cluster.local:6956";
+  private final String LOADBALANCER_PREFIX = "gobblinServiceLoadbalancer.";
+  private final String NUM_SCHEDULERS_KEY = "numSchedulers";
+  private final String FLOW_NAME_KEY = "flowName";
+  private final String FLOW_GROUP_KEY = "flowGroup";
+  private int numSchedulers;
+  protected final Logger _log;
+
+  ForwardRequestServlet(Config config, Optional log) {
+_log = log.isPresent() ? log.get() : LoggerFactory.getLogger(getClass());
+this.numSchedulers = config.getInt(LOADBALANCER_PREFIX + 
NUM_SCHEDULERS_KEY);
+_log.info("Load balancer started with {} servers", this.numSchedulers);
+this.client = HttpAsyncClients.createDefault();
 
 Review comment:
   We might be able to leverage `request.getRequestDispatcher` to forward 
request instead of using a client.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361982)
Remaining Estimate: 0h
Time Spent: 10m

> Create loadbalancer and shard FlowSpec schedules
> 
>
> Key: GOBBLIN-1009
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1009
> Project: Apache Gobblin
>  Issue Type: New Feature
>Reporter: William Lo
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Creates a load balancer to distribute incoming requests among multiple GaaS 
> instances.
> This allows there to be more than a single instance of GaaS running on a 
> kubernetes cluster without Apache Helix



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1009?focusedWorklogId=361984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361984
 ]

ASF GitHub Bot logged work on GOBBLIN-1009:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 00:37
Start Date: 21/Dec/19 00:37
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2854: [GOBBLIN-1009] 
Create load balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360614255
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/GobblinServiceLoadBalancer.java
 ##
 @@ -0,0 +1,57 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+import org.apache.gobblin.runtime.app.ApplicationException;
+import org.apache.gobblin.runtime.app.ApplicationLauncher;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class GobblinServiceLoadBalancer implements ApplicationLauncher {
+  private Server server;
 
 Review comment:
   Try if we can merge Load Balancing into the `nginx` server container. We 
might need create a load balancing image on top of an nginx image.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361984)
Time Spent: 20m  (was: 10m)

> Create loadbalancer and shard FlowSpec schedules
> 
>
> Key: GOBBLIN-1009
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1009
> Project: Apache Gobblin
>  Issue Type: New Feature
>Reporter: William Lo
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Creates a load balancer to distribute incoming requests among multiple GaaS 
> instances.
> This allows there to be more than a single instance of GaaS running on a 
> kubernetes cluster without Apache Helix



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

2019-12-20 Thread GitBox

zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load 
balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360615747
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/ForwardRequestServlet.java
 ##
 @@ -0,0 +1,174 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Enumeration;
+import java.util.concurrent.Future;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.ServletException;
+import org.apache.commons.io.IOUtils;
+import org.apache.gobblin.util.HashingUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
+import org.apache.http.impl.nio.client.HttpAsyncClients;
+import org.apache.http.protocol.HTTP;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ForwardRequestServlet extends HttpServlet {
+
+  public CloseableHttpAsyncClient client;
+
+  private final String STATEFULSET_URL_PREFIX = "http://gaas-;;
+  // TODO: split this up into configuration or something
+  private final String STATEFULSET_URL_SUFFIX = 
".gaas.default.svc.cluster.local:6956";
+  private final String LOADBALANCER_PREFIX = "gobblinServiceLoadbalancer.";
+  private final String NUM_SCHEDULERS_KEY = "numSchedulers";
+  private final String FLOW_NAME_KEY = "flowName";
+  private final String FLOW_GROUP_KEY = "flowGroup";
+  private int numSchedulers;
+  protected final Logger _log;
+
+  ForwardRequestServlet(Config config, Optional log) {
+_log = log.isPresent() ? log.get() : LoggerFactory.getLogger(getClass());
+this.numSchedulers = config.getInt(LOADBALANCER_PREFIX + 
NUM_SCHEDULERS_KEY);
+_log.info("Load balancer started with {} servers", this.numSchedulers);
+this.client = HttpAsyncClients.createDefault();
 
 Review comment:
   We might be able to leverage `request.getRequestDispatcher` to forward 
request instead of using a client.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

2019-12-20 Thread GitBox

zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load 
balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360614396
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/ForwardRequestServlet.java
 ##
 @@ -0,0 +1,174 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Enumeration;
+import java.util.concurrent.Future;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.ServletException;
+import org.apache.commons.io.IOUtils;
+import org.apache.gobblin.util.HashingUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
+import org.apache.http.impl.nio.client.HttpAsyncClients;
+import org.apache.http.protocol.HTTP;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ForwardRequestServlet extends HttpServlet {
+
+  public CloseableHttpAsyncClient client;
+
+  private final String STATEFULSET_URL_PREFIX = "http://gaas-;;
+  // TODO: split this up into configuration or something
+  private final String STATEFULSET_URL_SUFFIX = 
".gaas.default.svc.cluster.local:6956";
+  private final String LOADBALANCER_PREFIX = "gobblinServiceLoadbalancer.";
+  private final String NUM_SCHEDULERS_KEY = "numSchedulers";
+  private final String FLOW_NAME_KEY = "flowName";
+  private final String FLOW_GROUP_KEY = "flowGroup";
+  private int numSchedulers;
+  protected final Logger _log;
+
+  ForwardRequestServlet(Config config, Optional log) {
+_log = log.isPresent() ? log.get() : LoggerFactory.getLogger(getClass());
+this.numSchedulers = config.getInt(LOADBALANCER_PREFIX + 
NUM_SCHEDULERS_KEY);
+_log.info("Load balancer started with {} servers", this.numSchedulers);
+this.client = HttpAsyncClients.createDefault();
+this.client.start();
+  }
+
+  protected void doGet(HttpServletRequest request, HttpServletResponse 
response) {
 
 Review comment:
   Is this an `override`? If so, let's add `@override` annotation for better 
code readability, avoiding future readers asking this question. This one 
applies to the other methods


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

2019-12-20 Thread GitBox

zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load 
balancer for Gobblin Service
URL: https://github.com/apache/incubator-gobblin/pull/2854#discussion_r360614255
 
 

 ##
 File path: 
gobblin-service-loadbalancer/src/main/java/org/apache/gobblin/service/loadbalancer/GobblinServiceLoadBalancer.java
 ##
 @@ -0,0 +1,57 @@
+package org.apache.gobblin.service.loadbalancer;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+import org.apache.gobblin.runtime.app.ApplicationException;
+import org.apache.gobblin.runtime.app.ApplicationLauncher;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class GobblinServiceLoadBalancer implements ApplicationLauncher {
+  private Server server;
 
 Review comment:
   Try if we can merge Load Balancing into the `nginx` server container. We 
might need create a load balancing image on top of an nginx image.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-1010) Use Container Memory for Gobblin to run on pods

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1010?focusedWorklogId=361977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361977
 ]

ASF GitHub Bot logged work on GOBBLIN-1010:
---

Author: ASF GitHub Bot
Created on: 21/Dec/19 00:26
Start Date: 21/Dec/19 00:26
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2855: [GOBBLIN-1010] 
Adds flags to use cgroup memory for heap space available for Gobblin images
URL: https://github.com/apache/incubator-gobblin/pull/2855
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-1010
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   
   Currently Gobblin's memory usage is defined as arguments passed to its 
processes.
   In a containerized environment (i.e. Kubernetes), it is more accurate and 
usable if the process can just utilize the available memory in its container.
   
   This option is described in this article 
https://blog.softwaremill.com/docker-support-in-new-java-8-finally-fd595df0ca54
   
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361977)
Remaining Estimate: 0h
Time Spent: 10m

> Use Container Memory for Gobblin to run on pods
> ---
>
> Key: GOBBLIN-1010
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1010
> Project: Apache Gobblin
>  Issue Type: Improvement
>  Components: gobblin-service
>Reporter: William Lo
>Assignee: Abhishek Tiwari
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] Will-Lo opened a new pull request #2855: [GOBBLIN-1010] Adds flags to use cgroup memory for heap space available for Gobblin images

2019-12-20 Thread GitBox

Will-Lo opened a new pull request #2855: [GOBBLIN-1010] Adds flags to use 
cgroup memory for heap space available for Gobblin images
URL: https://github.com/apache/incubator-gobblin/pull/2855
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-1010
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   
   Currently Gobblin's memory usage is defined as arguments passed to its 
processes.
   In a containerized environment (i.e. Kubernetes), it is more accurate and 
usable if the process can just utilize the available memory in its container.
   
   This option is described in this article 
https://blog.softwaremill.com/docker-support-in-new-java-8-finally-fd595df0ca54
   
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (GOBBLIN-1010) Use Container Memory for Gobblin to run on pods

2019-12-20 Thread William Lo (Jira)

William Lo created GOBBLIN-1010:
---

 Summary: Use Container Memory for Gobblin to run on pods
 Key: GOBBLIN-1010
 URL: https://issues.apache.org/jira/browse/GOBBLIN-1010
 Project: Apache Gobblin
  Issue Type: Improvement
  Components: gobblin-service
Reporter: William Lo
Assignee: Abhishek Tiwari






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

2019-12-20 Thread William Lo (Jira)

William Lo created GOBBLIN-1009:
---

 Summary: Create loadbalancer and shard FlowSpec schedules
 Key: GOBBLIN-1009
 URL: https://issues.apache.org/jira/browse/GOBBLIN-1009
 Project: Apache Gobblin
  Issue Type: New Feature
Reporter: William Lo


Creates a load balancer to distribute incoming requests among multiple GaaS 
instances.
This allows there to be more than a single instance of GaaS running on a 
kubernetes cluster without Apache Helix



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] Will-Lo opened a new pull request #2854: Create load balancer

2019-12-20 Thread GitBox

Will-Lo opened a new pull request #2854: Create load balancer
URL: https://github.com/apache/incubator-gobblin/pull/2854
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-XXX
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if 
applicable):
   
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

2019-12-20 Thread GitBox

autumnust commented on a change in pull request #2846: [GOBBLIN-1001] Implement 
TimePartitionGlobFinder
URL: https://github.com/apache/incubator-gobblin/pull/2846#discussion_r360540204
 
 

 ##
 File path: 
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/dataset/SimpleFileSystemDataset.java
 ##
 @@ -46,6 +50,9 @@ public String datasetURN() {
 return path.toString();
   }
 
+  /**
+   * @return true if the dataset doesn't have a physical file/folder
+   */
   public boolean getIsVirtual() {
 
 Review comment:
   Does it sound more intuitive if `isVirtual` ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1001?focusedWorklogId=361823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361823
 ]

ASF GitHub Bot logged work on GOBBLIN-1001:
---

Author: ASF GitHub Bot
Created on: 20/Dec/19 19:34
Start Date: 20/Dec/19 19:34
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2846: 
[GOBBLIN-1001] Implement TimePartitionGlobFinder
URL: https://github.com/apache/incubator-gobblin/pull/2846#discussion_r360540204
 
 

 ##
 File path: 
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/dataset/SimpleFileSystemDataset.java
 ##
 @@ -46,6 +50,9 @@ public String datasetURN() {
 return path.toString();
   }
 
+  /**
+   * @return true if the dataset doesn't have a physical file/folder
+   */
   public boolean getIsVirtual() {
 
 Review comment:
   Does it sound more intuitive if `isVirtual` ? 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361823)
Time Spent: 3h 10m  (was: 3h)

> Implement TimePartitionGlobFinder
> -
>
> Key: GOBBLIN-1001
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1001
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-1001?focusedWorklogId=361821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361821
 ]

ASF GitHub Bot logged work on GOBBLIN-1001:
---

Author: ASF GitHub Bot
Created on: 20/Dec/19 19:31
Start Date: 20/Dec/19 19:31
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2846: [GOBBLIN-1001] 
Implement TimePartitionGlobFinder
URL: 
https://github.com/apache/incubator-gobblin/pull/2846#issuecomment-564186234
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=h1)
 Report
   > Merging 
[#2846](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/0a1debc30734ec57d2be7351270894756a6c18af?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `88.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2846  +/-   ##
   
   - Coverage 45.59%   45.57%   -0.03% 
   - Complexity 8984 9031  +47 
   
 Files  1904 1908   +4 
 Lines 7134771729 +382 
 Branches   7876 7912  +36 
   
   + Hits  3253432692 +158 
   - Misses3580636031 +225 
   + Partials   3007 3006   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...gement/copy/TimeAwareRecursiveCopyableDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZUF3YXJlUmVjdXJzaXZlQ29weWFibGVEYXRhc2V0LmphdmE=)
 | `91.17% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: |
   | 
[...anagement/dataset/DefaultFileSystemGlobFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvRGVmYXVsdEZpbGVTeXN0ZW1HbG9iRmluZGVyLmphdmE=)
 | `100% <100%> (+100%)` | `2 <1> (+2)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `63.15% <63.15%> (ø)` | `4 <4> (?)` | |
   | 
[...ta/management/dataset/SimpleFileSystemDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvU2ltcGxlRmlsZVN5c3RlbURhdGFzZXQuamF2YQ==)
 | `88.88% <88.88%> (ø)` | `4 <4> (?)` | |
   | 
[...ta/management/dataset/TimePartitionGlobFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvVGltZVBhcnRpdGlvbkdsb2JGaW5kZXIuamF2YQ==)
 | `93.4% <93.4%> (ø)` | `23 <23> (?)` | |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.81% <0%> (-2.95%)` | `12% <0%> (+1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-2.81%)` | `11% <0%> (-1%)` | |
   | 
[...service/modules/orchestration/DagManagerUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL0RhZ01hbmFnZXJVdGlscy5qYXZh)
 | `85.54% <0%> (-1.21%)` | `37% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==)
 | `14.84% <0%> (-0.85%)` | `3% <0%> (-1%)` | |
   |

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

2019-12-20 Thread GitBox

codecov-io edited a comment on issue #2846: [GOBBLIN-1001] Implement 
TimePartitionGlobFinder
URL: 
https://github.com/apache/incubator-gobblin/pull/2846#issuecomment-564186234
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=h1)
 Report
   > Merging 
[#2846](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/0a1debc30734ec57d2be7351270894756a6c18af?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `88.33%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2846  +/-   ##
   
   - Coverage 45.59%   45.57%   -0.03% 
   - Complexity 8984 9031  +47 
   
 Files  1904 1908   +4 
 Lines 7134771729 +382 
 Branches   7876 7912  +36 
   
   + Hits  3253432692 +158 
   - Misses3580636031 +225 
   + Partials   3007 3006   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2846?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...gement/copy/TimeAwareRecursiveCopyableDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZUF3YXJlUmVjdXJzaXZlQ29weWFibGVEYXRhc2V0LmphdmE=)
 | `91.17% <ø> (ø)` | `15 <0> (ø)` | :arrow_down: |
   | 
[...anagement/dataset/DefaultFileSystemGlobFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvRGVmYXVsdEZpbGVTeXN0ZW1HbG9iRmluZGVyLmphdmE=)
 | `100% <100%> (+100%)` | `2 <1> (+2)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/time/TimeIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdGltZS9UaW1lSXRlcmF0b3IuamF2YQ==)
 | `63.15% <63.15%> (ø)` | `4 <4> (?)` | |
   | 
[...ta/management/dataset/SimpleFileSystemDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvU2ltcGxlRmlsZVN5c3RlbURhdGFzZXQuamF2YQ==)
 | `88.88% <88.88%> (ø)` | `4 <4> (?)` | |
   | 
[...ta/management/dataset/TimePartitionGlobFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvVGltZVBhcnRpdGlvbkdsb2JGaW5kZXIuamF2YQ==)
 | `93.4% <93.4%> (ø)` | `23 <23> (?)` | |
   | 
[...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh)
 | `19.81% <0%> (-2.95%)` | `12% <0%> (+1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-2.81%)` | `11% <0%> (-1%)` | |
   | 
[...service/modules/orchestration/DagManagerUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL0RhZ01hbmFnZXJVdGlscy5qYXZh)
 | `85.54% <0%> (-1.21%)` | `37% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==)
 | `14.84% <0%> (-0.85%)` | `3% <0%> (-1%)` | |
   | 
[...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=)
 | `64.51% <0%> (-0.27%)` | `85% <0%> (+1%)` | |
   | ... and [31 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2846/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at

[jira] [Resolved] (GOBBLIN-988) Create Local FS Job Status Retriever

2019-12-20 Thread Hung Tran (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hung Tran resolved GOBBLIN-988.
---
Fix Version/s: 0.15.0
   Resolution: Fixed

Issue resolved by pull request #2834
[https://github.com/apache/incubator-gobblin/pull/2834]

> Create Local FS Job Status Retriever
> 
>
> Key: GOBBLIN-988
> URL: https://issues.apache.org/jira/browse/GOBBLIN-988
> Project: Apache Gobblin
>  Issue Type: New Feature
>Reporter: William Lo
>Priority: Minor
> Fix For: 0.15.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> GaaS currently only supports Kafka to monitor job status in the dagManager.
> Create a simple jobstatusmonitor that can track completed jobs in gobblin 
> standalone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-988) Create Local FS Job Status Retriever

2019-12-20 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-988?focusedWorklogId=361790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361790
 ]

ASF GitHub Bot logged work on GOBBLIN-988:
--

Author: ASF GitHub Bot
Created on: 20/Dec/19 18:55
Start Date: 20/Dec/19 18:55
Worklog Time Spent: 10m 
  Work Description: asfgit commented on pull request #2834: [GOBBLIN-988] 
Implement LocalFSJobStatusRetriever
URL: https://github.com/apache/incubator-gobblin/pull/2834
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 361790)
Time Spent: 1h 50m  (was: 1h 40m)

> Create Local FS Job Status Retriever
> 
>
> Key: GOBBLIN-988
> URL: https://issues.apache.org/jira/browse/GOBBLIN-988
> Project: Apache Gobblin
>  Issue Type: New Feature
>Reporter: William Lo
>Priority: Minor
> Fix For: 0.15.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> GaaS currently only supports Kafka to monitor job status in the dagManager.
> Create a simple jobstatusmonitor that can track completed jobs in gobblin 
> standalone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] asfgit closed pull request #2834: [GOBBLIN-988] Implement LocalFSJobStatusRetriever

2019-12-20 Thread GitBox

asfgit closed pull request #2834: [GOBBLIN-988] Implement 
LocalFSJobStatusRetriever
URL: https://github.com/apache/incubator-gobblin/pull/2834
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

[GitHub] [incubator-gobblin] codecov-io commented on issue #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

[jira] [Work logged] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

[GitHub] [incubator-gobblin] zxcware opened a new pull request #2856: [GOBBLIN-1011] adjust compaction flow to work with virtual partition

[jira] [Updated] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

[jira] [Created] (GOBBLIN-1011) Adjust compaction flow to work with virtual partition

[jira] [Resolved] (GOBBLIN-1001) Implement TimePartitionGlobFinder

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

[GitHub] [incubator-gobblin] asfgit closed pull request #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

[jira] [Work logged] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2854: [GOBBLIN-1009] Create load balancer for Gobblin Service

[jira] [Work logged] (GOBBLIN-1010) Use Container Memory for Gobblin to run on pods

[GitHub] [incubator-gobblin] Will-Lo opened a new pull request #2855: [GOBBLIN-1010] Adds flags to use cgroup memory for heap space available for Gobblin images

[jira] [Created] (GOBBLIN-1010) Use Container Memory for Gobblin to run on pods

[jira] [Created] (GOBBLIN-1009) Create loadbalancer and shard FlowSpec schedules

[GitHub] [incubator-gobblin] Will-Lo opened a new pull request #2854: Create load balancer

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

[jira] [Work logged] (GOBBLIN-1001) Implement TimePartitionGlobFinder

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2846: [GOBBLIN-1001] Implement TimePartitionGlobFinder

[jira] [Resolved] (GOBBLIN-988) Create Local FS Job Status Retriever

[jira] [Work logged] (GOBBLIN-988) Create Local FS Job Status Retriever

[GitHub] [incubator-gobblin] asfgit closed pull request #2834: [GOBBLIN-988] Implement LocalFSJobStatusRetriever

29 matches

Site Navigation

Mail list logo

Footer information