[
https://issues.apache.org/jira/browse/HUDI-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937336#comment-16937336
]
Xing Pan commented on HUDI-269:
-------------------------------
and I ran delta streamer like:
{code:java}
spark-submit --class
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
--executor-memory 1g \
--executor-cores 1 \
--conf spark.dynamicAllocation.executorIdleTimeout=10s \
--conf spark.dynamicAllocation.maxExecutors=3 \
xxx/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar \
--target-base-path s3a://xxx \
--target-table default.xxx \
--storage-type MERGE_ON_READ \
--props s3a://xxxx/kafka-source.properties \
--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
--source-ordering-field ts \
--schemaprovider-class
org.apache.hudi.utilities.schema.SchemaRegistryProvider \
--op UPSERT \
--enable-hive-sync \
--continuous \
--min-sync-interval-seconds 0 \
--hoodie-conf hoodie.embed.timeline.server=true \
--hoodie-conf hoodie.filesystem.view.incr.timeline.sync.enable=true
{code}
> Provide ability to throttle DeltaStreamer sync runs
> ---------------------------------------------------
>
> Key: HUDI-269
> URL: https://issues.apache.org/jira/browse/HUDI-269
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: deltastreamer
> Reporter: BALAJI VARADARAJAN
> Assignee: Xing Pan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: image-2019-09-25-08-51-19-686.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Copied from [https://github.com/apache/incubator-hudi/issues/922]
> In some scenario in our cluster, we may want delta streamer to slow down a
> bit.
> so it's nice to have a parameter to control the min sync interval of each
> sync in continuous mode.
> this param is default to 0, so this should not affect current logic.
> minor pr: [#921|https://github.com/apache/incubator-hudi/pull/921]
> the main reason we want to slow it down is that aws s3 is charged by s3
> get/put/list requests. we don't want to pay for too many requests for a
> really slow change table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
