[ https://issues.apache.org/jira/browse/GOBBLIN-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kuai Yu reassigned GOBBLIN-19: ------------------------------ Assignee: Kuai Yu Sprint: Apache Gobblin 170807 > dataset specific properties are ignored&dropped by KafkaBiLevelWorkUnitPacker > ----------------------------------------------------------------------------- > > Key: GOBBLIN-19 > URL: https://issues.apache.org/jira/browse/GOBBLIN-19 > Project: Apache Gobblin > Issue Type: Bug > Reporter: Clemens Valiente > Assignee: Kuai Yu > > I failed to get dataset.specific.props to work on our jobs, and I think I > found the reason: > in KafkaSource.getWorkUnitForTopicPartition the properties are added > correctly to the individual workunits. > The KafkaBiLevelWorkUnitPacker then assigns the WorkUnits to their bins and > combines them into one WorkUnit in squeezeMultiWorkUnit() but doesn't copy > over the topicSpecificSettings. > Using the KafkaSingleLevelWorkUnitPacker works fine with > dataset.specific.props since it doesn't call squeezeMultiWorkUnit on > non-empty workUnits. > > *Github Url* : https://github.com/linkedin/gobblin/issues/1901 > *Github Reporter* : [~cvaliente] > *Github Created At* : 2017-05-26T09:25:54Z > *Github Updated At* : 2017-05-31T06:39:04Z > h3. Comments > ---- > [~cvaliente] wrote on 2017-05-26T10:55:37Z : fix in #1903 > > *Github Url* : > https://github.com/linkedin/gobblin/issues/1901#issuecomment-304253329 > ---- > [~stakiar] wrote on 2017-05-30T17:42:07Z : Doesn't > `KafkaSource#addTopicSpecificPropsToWorkUnits` handle adding dataset specific > configuration? That method is run after the bin-packing is done. So if > `dataset.specific.props` isn't working I would guess the bug would be in that > method. > > *Github Url* : > https://github.com/linkedin/gobblin/issues/1901#issuecomment-304953996 > ---- > [~cvaliente] wrote on 2017-05-31T06:39:04Z : You are right, that wasn't yet > implemented in 0.9 and I forgot to check upstream. > > *Github Url* : > https://github.com/linkedin/gobblin/issues/1901#issuecomment-305098396 -- This message was sent by Atlassian JIRA (v6.4.14#64029)