[GitHub] [incubator-gobblin] codecov-io commented on issue #2905: [GOBBLIN-1064]Make KafkaAvroSchemaRegistry extendable
codecov-io commented on issue #2905: [GOBBLIN-1064]Make KafkaAvroSchemaRegistry extendable URL: https://github.com/apache/incubator-gobblin/pull/2905#issuecomment-593727502 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=h1) Report > Merging [#2905](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/998c968cb84ff7a0f3951ae648cb9d2ed1c8e4ad?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2905 +/- ## - Coverage 45.86% 45.85% -0.02% - Complexity 9169 9181 +12 Files 1932 1933 +1 Lines 7279572836 +41 Branches 8022 8030 +8 + Hits 3338933400 +11 - Misses3634936374 +25 - Partials 3057 3062 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/gobblin/runtime/fork/Fork.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvZm9yay9Gb3JrLmphdmE=) | `75% <ø> (ø)` | `68 <0> (ø)` | :arrow_down: | | [...gobblin/metrics/kafka/KafkaAvroSchemaRegistry.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FBdnJvU2NoZW1hUmVnaXN0cnkuamF2YQ==) | `14.81% <ø> (ø)` | `2 <0> (ø)` | :arrow_down: | | [.../hive/filter/LookbackPartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvTG9va2JhY2tQYXJ0aXRpb25GaWx0ZXJHZW5lcmF0b3IuamF2YQ==) | `78.94% <0%> (-21.06%)` | `6% <0%> (+3%)` | | | [...re/ControllerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0NvbnRyb2xsZXJVc2VyRGVmaW5lZE1lc3NhZ2VIYW5kbGVyRmFjdG9yeS5qYXZh) | `39.06% <0%> (-12.5%)` | `4% <0%> (ø)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...estli/GobblinServiceFlowConfigResourceHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9yZXN0bGkvR29iYmxpblNlcnZpY2VGbG93Q29uZmlnUmVzb3VyY2VIYW5kbGVyLmphdmE=) | `44.61% <0%> (-6.16%)` | `10% <0%> (-1%)` | | | [.../modules/scheduler/GobblinServiceJobScheduler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9zY2hlZHVsZXIvR29iYmxpblNlcnZpY2VKb2JTY2hlZHVsZXIuamF2YQ==) | `53.71% <0%> (-4.58%)` | `19% <0%> (-2%)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...rg/apache/gobblin/service/FlowConfigsResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1Jlc291cmNlLmphdmE=) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...hive/filter/DateRangePartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvRGF0ZVJhbmdlUGFydGl0aW9uRmlsdGVyR2VuZXJhdG9yLmphdmE=)
[jira] [Work logged] (GOBBLIN-1064) Make KafkaAvroSchemaRegistry extendable
[ https://issues.apache.org/jira/browse/GOBBLIN-1064?focusedWorklogId=396557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396557 ] ASF GitHub Bot logged work on GOBBLIN-1064: --- Author: ASF GitHub Bot Created on: 03/Mar/20 02:03 Start Date: 03/Mar/20 02:03 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2905: [GOBBLIN-1064]Make KafkaAvroSchemaRegistry extendable URL: https://github.com/apache/incubator-gobblin/pull/2905#issuecomment-593727502 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=h1) Report > Merging [#2905](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/998c968cb84ff7a0f3951ae648cb9d2ed1c8e4ad?src=pr=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2905 +/- ## - Coverage 45.86% 45.85% -0.02% - Complexity 9169 9181 +12 Files 1932 1933 +1 Lines 7279572836 +41 Branches 8022 8030 +8 + Hits 3338933400 +11 - Misses3634936374 +25 - Partials 3057 3062 +5 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2905?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/gobblin/runtime/fork/Fork.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvZm9yay9Gb3JrLmphdmE=) | `75% <ø> (ø)` | `68 <0> (ø)` | :arrow_down: | | [...gobblin/metrics/kafka/KafkaAvroSchemaRegistry.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FBdnJvU2NoZW1hUmVnaXN0cnkuamF2YQ==) | `14.81% <ø> (ø)` | `2 <0> (ø)` | :arrow_down: | | [.../hive/filter/LookbackPartitionFilterGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9maWx0ZXIvTG9va2JhY2tQYXJ0aXRpb25GaWx0ZXJHZW5lcmF0b3IuamF2YQ==) | `78.94% <0%> (-21.06%)` | `6% <0%> (+3%)` | | | [...re/ControllerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0NvbnRyb2xsZXJVc2VyRGVmaW5lZE1lc3NhZ2VIYW5kbGVyRmFjdG9yeS5qYXZh) | `39.06% <0%> (-12.5%)` | `4% <0%> (ø)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...estli/GobblinServiceFlowConfigResourceHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9yZXN0bGkvR29iYmxpblNlcnZpY2VGbG93Q29uZmlnUmVzb3VyY2VIYW5kbGVyLmphdmE=) | `44.61% <0%> (-6.16%)` | `10% <0%> (-1%)` | | | [.../modules/scheduler/GobblinServiceJobScheduler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9zY2hlZHVsZXIvR29iYmxpblNlcnZpY2VKb2JTY2hlZHVsZXIuamF2YQ==) | `53.71% <0%> (-4.58%)` | `19% <0%> (-2%)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2905/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | |
[jira] [Updated] (GOBBLIN-1064) Make KafkaAvroSchemaRegistry extendable
[ https://issues.apache.org/jira/browse/GOBBLIN-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihan Li updated GOBBLIN-1064: -- Summary: Make KafkaAvroSchemaRegistry extendable (was: Add writer's schema to workUnitState) > Make KafkaAvroSchemaRegistry extendable > --- > > Key: GOBBLIN-1064 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1064 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1064) Add writer's schema to workUnitState
[ https://issues.apache.org/jira/browse/GOBBLIN-1064?focusedWorklogId=396406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396406 ] ASF GitHub Bot logged work on GOBBLIN-1064: --- Author: ASF GitHub Bot Created on: 02/Mar/20 21:45 Start Date: 02/Mar/20 21:45 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on issue #2905: [GOBBLIN-1064]Add writer schema to WorkUnitState URL: https://github.com/apache/incubator-gobblin/pull/2905#issuecomment-593639171 @autumnust Can you take a look at this minor change? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 396406) Time Spent: 20m (was: 10m) > Add writer's schema to workUnitState > > > Key: GOBBLIN-1064 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1064 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 commented on issue #2905: [GOBBLIN-1064]Add writer schema to WorkUnitState
ZihanLi58 commented on issue #2905: [GOBBLIN-1064]Add writer schema to WorkUnitState URL: https://github.com/apache/incubator-gobblin/pull/2905#issuecomment-593639171 @autumnust Can you take a look at this minor change? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1040) Fix High level consumer
[
https://issues.apache.org/jira/browse/GOBBLIN-1040?focusedWorklogId=396282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396282
]
ASF GitHub Bot logged work on GOBBLIN-1040:
---
Author: ASF GitHub Bot
Created on: 02/Mar/20 18:27
Start Date: 02/Mar/20 18:27
Worklog Time Spent: 10m
Work Description: vikrambohra commented on pull request #2900:
[GOBBLIN-1040] HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386568726
##
File path:
gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/service/StreamingKafkaSpecExecutorTest.java
##
@@ -183,9 +196,4 @@ public void after() {
cleanupTestDir();
}
-
Review comment:
yes, I moved this test class to gobblin-kafka-09
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 396282)
Time Spent: 3.5h (was: 3h 20m)
> Fix High level consumer
>
>
> Key: GOBBLIN-1040
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1040
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Vikram Bohra
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] vikrambohra commented on a change in pull request #2900: [GOBBLIN-1040] HighLevelConsumer re-design by removing references to …
vikrambohra commented on a change in pull request #2900: [GOBBLIN-1040]
HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386568726
##
File path:
gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/service/StreamingKafkaSpecExecutorTest.java
##
@@ -183,9 +196,4 @@ public void after() {
cleanupTestDir();
}
-
Review comment:
yes, I moved this test class to gobblin-kafka-09
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1064) Add writer's schema to workUnitState
[
https://issues.apache.org/jira/browse/GOBBLIN-1064?focusedWorklogId=396280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396280
]
ASF GitHub Bot logged work on GOBBLIN-1064:
---
Author: ASF GitHub Bot
Created on: 02/Mar/20 18:26
Start Date: 02/Mar/20 18:26
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on pull request #2905:
[GOBBLIN-1064]Add writer schema to WorkUnitState
URL: https://github.com/apache/incubator-gobblin/pull/2905
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-1064] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1064
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Add the writer schema for to WorkUnitState
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Make sure at the publish level, workUnitState contains the writer schema
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 396280)
Remaining Estimate: 0h
Time Spent: 10m
> Add writer's schema to workUnitState
>
>
> Key: GOBBLIN-1064
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1064
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 opened a new pull request #2905: [GOBBLIN-1064]Add writer schema to WorkUnitState
ZihanLi58 opened a new pull request #2905: [GOBBLIN-1064]Add writer schema to
WorkUnitState
URL: https://github.com/apache/incubator-gobblin/pull/2905
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-1064] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1064
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Add the writer schema for to WorkUnitState
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Make sure at the publish level, workUnitState contains the writer schema
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1040) Fix High level consumer
[
https://issues.apache.org/jira/browse/GOBBLIN-1040?focusedWorklogId=396265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396265
]
ASF GitHub Bot logged work on GOBBLIN-1040:
---
Author: ASF GitHub Bot
Created on: 02/Mar/20 17:59
Start Date: 02/Mar/20 17:59
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2900: [GOBBLIN-1040]
HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386554251
##
File path:
gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java
##
@@ -160,14 +164,30 @@ public long getLatestOffset(KafkaPartition partition)
throws KafkaOffsetRetrieva
this.consumer.assign(Lists.newArrayList(new
TopicPartition(partition.getTopicName(), partition.getId(;
this.consumer.seek(new TopicPartition(partition.getTopicName(),
partition.getId()), nextOffset);
-ConsumerRecords consumerRecords =
consumer.poll(super.fetchTimeoutMillis);
-return Iterators.transform(consumerRecords.iterator(), new
Function, KafkaConsumerRecord>() {
+return consume();
+ }
- @Override
- public KafkaConsumerRecord apply(ConsumerRecord input) {
-return new Kafka09ConsumerRecord<>(input);
- }
-});
+ @Override
+ public synchronized Iterator consume() {
Review comment:
I understand. My point is our implementation should ensure that we are
always calling consume(), commitAsycn() from a single thread, which I believe
to be the case in this PR. Putting a synchronized on this method would simply
hide an implementation bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 396265)
Time Spent: 3h 20m (was: 3h 10m)
> Fix High level consumer
>
>
> Key: GOBBLIN-1040
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1040
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Vikram Bohra
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2900: [GOBBLIN-1040] HighLevelConsumer re-design by removing references to …
sv2000 commented on a change in pull request #2900: [GOBBLIN-1040]
HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386554251
##
File path:
gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java
##
@@ -160,14 +164,30 @@ public long getLatestOffset(KafkaPartition partition)
throws KafkaOffsetRetrieva
this.consumer.assign(Lists.newArrayList(new
TopicPartition(partition.getTopicName(), partition.getId(;
this.consumer.seek(new TopicPartition(partition.getTopicName(),
partition.getId()), nextOffset);
-ConsumerRecords consumerRecords =
consumer.poll(super.fetchTimeoutMillis);
-return Iterators.transform(consumerRecords.iterator(), new
Function, KafkaConsumerRecord>() {
+return consume();
+ }
- @Override
- public KafkaConsumerRecord apply(ConsumerRecord input) {
-return new Kafka09ConsumerRecord<>(input);
- }
-});
+ @Override
+ public synchronized Iterator consume() {
Review comment:
I understand. My point is our implementation should ensure that we are
always calling consume(), commitAsycn() from a single thread, which I believe
to be the case in this PR. Putting a synchronized on this method would simply
hide an implementation bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1040) Fix High level consumer
[
https://issues.apache.org/jira/browse/GOBBLIN-1040?focusedWorklogId=396259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396259
]
ASF GitHub Bot logged work on GOBBLIN-1040:
---
Author: ASF GitHub Bot
Created on: 02/Mar/20 17:47
Start Date: 02/Mar/20 17:47
Worklog Time Spent: 10m
Work Description: vikrambohra commented on pull request #2900:
[GOBBLIN-1040] HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386547675
##
File path:
gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java
##
@@ -160,14 +164,30 @@ public long getLatestOffset(KafkaPartition partition)
throws KafkaOffsetRetrieva
this.consumer.assign(Lists.newArrayList(new
TopicPartition(partition.getTopicName(), partition.getId(;
this.consumer.seek(new TopicPartition(partition.getTopicName(),
partition.getId()), nextOffset);
-ConsumerRecords consumerRecords =
consumer.poll(super.fetchTimeoutMillis);
-return Iterators.transform(consumerRecords.iterator(), new
Function, KafkaConsumerRecord>() {
+return consume();
+ }
- @Override
- public KafkaConsumerRecord apply(ConsumerRecord input) {
-return new Kafka09ConsumerRecord<>(input);
- }
-});
+ @Override
+ public synchronized Iterator consume() {
Review comment:
Underlying Kafka Consumer is not thread safe. consume() commitAsync(),
committed() all try to acquire a lock (and they are different threads) the
consumer throws a ConcurrentModificationException if it detects that the lock
has been acquired by another thread.
/**
* Acquire the light lock protecting this consumer from multi-threaded
access. Instead of blocking
* when the lock is not available, however, we just throw an exception
(since multi-threaded usage is not
* supported).
* @throws IllegalStateException if the consumer has been closed
* @throws ConcurrentModificationException if another thread already has
the lock
*/
private void acquire() {
ensureNotClosed();
long threadId = Thread.currentThread().getId();
if (threadId != currentThread.get() &&
!currentThread.compareAndSet(NO_CURRENT_THREAD, threadId))
throw new ConcurrentModificationException("KafkaConsumer is not
safe for multi-threaded access");
refcount.incrementAndGet();
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 396259)
Time Spent: 3h 10m (was: 3h)
> Fix High level consumer
>
>
> Key: GOBBLIN-1040
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1040
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Vikram Bohra
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] vikrambohra commented on a change in pull request #2900: [GOBBLIN-1040] HighLevelConsumer re-design by removing references to …
vikrambohra commented on a change in pull request #2900: [GOBBLIN-1040]
HighLevelConsumer re-design by removing references to …
URL: https://github.com/apache/incubator-gobblin/pull/2900#discussion_r386547675
##
File path:
gobblin-modules/gobblin-kafka-09/src/main/java/org/apache/gobblin/kafka/client/Kafka09ConsumerClient.java
##
@@ -160,14 +164,30 @@ public long getLatestOffset(KafkaPartition partition)
throws KafkaOffsetRetrieva
this.consumer.assign(Lists.newArrayList(new
TopicPartition(partition.getTopicName(), partition.getId(;
this.consumer.seek(new TopicPartition(partition.getTopicName(),
partition.getId()), nextOffset);
-ConsumerRecords consumerRecords =
consumer.poll(super.fetchTimeoutMillis);
-return Iterators.transform(consumerRecords.iterator(), new
Function, KafkaConsumerRecord>() {
+return consume();
+ }
- @Override
- public KafkaConsumerRecord apply(ConsumerRecord input) {
-return new Kafka09ConsumerRecord<>(input);
- }
-});
+ @Override
+ public synchronized Iterator consume() {
Review comment:
Underlying Kafka Consumer is not thread safe. consume() commitAsync(),
committed() all try to acquire a lock (and they are different threads) the
consumer throws a ConcurrentModificationException if it detects that the lock
has been acquired by another thread.
/**
* Acquire the light lock protecting this consumer from multi-threaded
access. Instead of blocking
* when the lock is not available, however, we just throw an exception
(since multi-threaded usage is not
* supported).
* @throws IllegalStateException if the consumer has been closed
* @throws ConcurrentModificationException if another thread already has
the lock
*/
private void acquire() {
ensureNotClosed();
long threadId = Thread.currentThread().getId();
if (threadId != currentThread.get() &&
!currentThread.compareAndSet(NO_CURRENT_THREAD, threadId))
throw new ConcurrentModificationException("KafkaConsumer is not
safe for multi-threaded access");
refcount.incrementAndGet();
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (GOBBLIN-1066) field projection with namespace
[ https://issues.apache.org/jira/browse/GOBBLIN-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-1066. Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request #2904 [https://github.com/apache/incubator-gobblin/pull/2904] > field projection with namespace > --- > > Key: GOBBLIN-1066 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1066 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen >Priority: Major > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > `AvroProjectionConverter` currently ignores extract namespace to identify > fields to remove for a table. The change is to identify fields to remove with > namespace into account, configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2904: [GOBBLIN-1066] field projection with namespace
asfgit closed pull request #2904: [GOBBLIN-1066] field projection with namespace URL: https://github.com/apache/incubator-gobblin/pull/2904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1066) field projection with namespace
[ https://issues.apache.org/jira/browse/GOBBLIN-1066?focusedWorklogId=396254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396254 ] ASF GitHub Bot logged work on GOBBLIN-1066: --- Author: ASF GitHub Bot Created on: 02/Mar/20 17:33 Start Date: 02/Mar/20 17:33 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2904: [GOBBLIN-1066] field projection with namespace URL: https://github.com/apache/incubator-gobblin/pull/2904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 396254) Time Spent: 20m (was: 10m) > field projection with namespace > --- > > Key: GOBBLIN-1066 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1066 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > `AvroProjectionConverter` currently ignores extract namespace to identify > fields to remove for a table. The change is to identify fields to remove with > namespace into account, configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
