[ https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948393#comment-16948393 ]
Sean R. Owen commented on SPARK-29424: -------------------------------------- I doubt we want to throw yet another limit/config at this. It's hard to guess or impose a _task_ limit in order to limit cluster usage. This is often what resource constrains on the resource manager are for, not also duplicated in Spark. > Prevent Spark to committing stage of too much Task > -------------------------------------------------- > > Key: SPARK-29424 > URL: https://issues.apache.org/jira/browse/SPARK-29424 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.4.0, 3.0.0 > Reporter: angerszhu > Priority: Major > > Our user always submit bad SQL in query platform, Such as : > # write wrong join condition but submit that sql > # write wrong where condition > # etc.. > This case will make Spark scheduler to submit a lot of task. It will cause > spark run very slow and impact other user(spark thrift server) even run out > of memory because of too many object generated by a big num of tasks. > So I add a constraint when submit tasks and abort stage early when TaskSet > size num is bigger then set limit . I wonder if the community will accept > this way. > cc [~srowen] [~dongjoon] [~yumwang] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org