Thomas Graves created SPARK-27760: ------------------------------------- Summary: Spark resources - user configs change .count to be .amount, and yarn configs should match Key: SPARK-27760 URL: https://issues.apache.org/jira/browse/SPARK-27760 Project: Spark Issue Type: Story Components: Spark Core Affects Versions: 3.0.0 Reporter: Thomas Graves
For the Spark resources, we created the config spark.\{driver/executor}.resource.\{resourceName}.count I think we should change .count to be .amount. That more easily allows users to specify things with suffix like memory in a single config and they can combine the value and unit. Without this they would have to specify 2 separate configs which seems more of a hassle for the user. Spark on yarn already had added configs to request resources via the configs spark.yarn.\{executor/driver/am}.resource=<some amount>, where the <some amont> is value and unit together. We should change those configs to have a .amount suffix on them to match the spark configs and to allow future configs to be more easily added. YARN itself already supports tags and attributes so if we want the user to be able to pass those from spark at some point having a suffix makes sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org