[
https://issues.apache.org/jira/browse/GOBBLIN-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuai Yu reassigned GOBBLIN-19:
------------------------------
Assignee: Kuai Yu
Sprint: Apache Gobblin 170807
> dataset specific properties are ignored&dropped by KafkaBiLevelWorkUnitPacker
> -----------------------------------------------------------------------------
>
> Key: GOBBLIN-19
> URL: https://issues.apache.org/jira/browse/GOBBLIN-19
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Clemens Valiente
> Assignee: Kuai Yu
>
> I failed to get dataset.specific.props to work on our jobs, and I think I
> found the reason:
> in KafkaSource.getWorkUnitForTopicPartition the properties are added
> correctly to the individual workunits.
> The KafkaBiLevelWorkUnitPacker then assigns the WorkUnits to their bins and
> combines them into one WorkUnit in squeezeMultiWorkUnit() but doesn't copy
> over the topicSpecificSettings.
> Using the KafkaSingleLevelWorkUnitPacker works fine with
> dataset.specific.props since it doesn't call squeezeMultiWorkUnit on
> non-empty workUnits.
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1901
> *Github Reporter* : [~cvaliente]
> *Github Created At* : 2017-05-26T09:25:54Z
> *Github Updated At* : 2017-05-31T06:39:04Z
> h3. Comments
> ----
> [~cvaliente] wrote on 2017-05-26T10:55:37Z : fix in #1903
>
> *Github Url* :
> https://github.com/linkedin/gobblin/issues/1901#issuecomment-304253329
> ----
> [~stakiar] wrote on 2017-05-30T17:42:07Z : Doesn't
> `KafkaSource#addTopicSpecificPropsToWorkUnits` handle adding dataset specific
> configuration? That method is run after the bin-packing is done. So if
> `dataset.specific.props` isn't working I would guess the bug would be in that
> method.
>
> *Github Url* :
> https://github.com/linkedin/gobblin/issues/1901#issuecomment-304953996
> ----
> [~cvaliente] wrote on 2017-05-31T06:39:04Z : You are right, that wasn't yet
> implemented in 0.9 and I forgot to check upstream.
>
> *Github Url* :
> https://github.com/linkedin/gobblin/issues/1901#issuecomment-305098396
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
