[
https://issues.apache.org/jira/browse/HUDI-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978324#comment-16978324
]
leesf commented on HUDI-288:
----------------------------
> I think we can stick to the same whitelist/blacklist that Kafka itself uses?
It makes sense.
> IIUC, even now, we can specifiy multiple topics as source but they get
>written as a single Hudi dataset.
Take a look to the currently code, and find the config
_hoodie.deltastreamer.source.kafka.topic_ to identify the topic to ingest, and
I think it does not support topics, so we only support configuring only one
topic to ingest currently, any thing I missed and please correct me if I am
wrong.
> we want to ingest kafka topics are separate Hudi datasets. 1-1 mapping
>between a kafka topic and a hudi dataset.. I think the tool can take a
>`--base-path-prefix` and place each hudi dataset under
>`<base_path_prefix>/<topic_name>`
It makes sense.
> Also we could allow topic level overrides as needed.. for deltra steamer/hudi
>properties.. Our DFSPropertiesConfiguration class already supports includes as
>well.
Sorry not to understand it correctly. Could you please show more details?
> Are you targetting this for 0.5.1 next release? Or do you think we can pick
>up some things already labelled for that release.
I would like to get it ready for 0.5.1.
> Add support for ingesting multiple kafka streams in a single DeltaStreamer
> deployment
> -------------------------------------------------------------------------------------
>
> Key: HUDI-288
> URL: https://issues.apache.org/jira/browse/HUDI-288
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: deltastreamer
> Reporter: Vinoth Chandar
> Assignee: leesf
> Priority: Major
>
> https://lists.apache.org/thread.html/3a69934657c48b1c0d85cba223d69cb18e18cd8aaa4817c9fd72cef6@<dev.hudi.apache.org>
> has all the context
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
