[
https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850137#comment-16850137
]
Stavros Kontopoulos edited comment on SPARK-24815 at 5/28/19 8:43 PM:
----------------------------------------------------------------------
That is great news [~Karthik Palaniappan], I was thinking the same but missed
this ticket. So please make sure you add a section with current drawbacks of
using the batch mode dynamic allocation and why this feature is needed. Let me
know If I can help in any way.
was (Author: skonto):
That is great news [~Karthik Palaniappan], I was thinking the same but missed
this ticket. So please make sure you add a section with current drawbacks of
using the batch mode dynamic allocation and why this feature is needed.
> Structured Streaming should support dynamic allocation
> ------------------------------------------------------
>
> Key: SPARK-24815
> URL: https://issues.apache.org/jira/browse/SPARK-24815
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Structured Streaming
> Affects Versions: 2.3.1
> Reporter: Karthik Palaniappan
> Priority: Minor
>
> Dynamic allocation is very useful for adding and removing containers to match
> the actual workload. On multi-tenant clusters, it ensures that a Spark job is
> taking no more resources than necessary. In cloud environments, it enables
> autoscaling.
> However, if you set spark.dynamicAllocation.enabled=true and run a structured
> streaming job, Core's dynamic allocation algorithm kicks in. It requests
> executors if the task backlog is a certain size, and remove executors if they
> idle for a certain period of time.
> This does not make sense for streaming jobs, as outlined in
> https://issues.apache.org/jira/browse/SPARK-12133, which introduced dynamic
> allocation for the old streaming API.
> First, Spark should print a warning if you run a structured streaming job
> when Core's dynamic allocation is enabled
> Second, structured streaming should have support for dynamic allocation. It
> would be convenient if it were the same set of properties as Core's dynamic
> allocation, but I don't have a strong opinion on that.
> If somebody can give me pointers on how to add dynamic allocation support,
> I'd be happy to take a stab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
