[ https://issues.apache.org/jira/browse/SPARK-32221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-32221: ------------------------------------ Assignee: Apache Spark > Avoid possible errors due to incorrect file size or type supplied in spark > conf. > -------------------------------------------------------------------------------- > > Key: SPARK-32221 > URL: https://issues.apache.org/jira/browse/SPARK-32221 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes > Affects Versions: 3.1.0 > Reporter: Prashant Sharma > Assignee: Apache Spark > Priority: Major > > This would avoid failures, in case the files are a bit large or a user places > a binary file inside the SPARK_CONF_DIR. > Both of which are not supported at the moment. > The reason is, underlying etcd store does limit the size of each entry to > only 1 MiB( Recent versions of K8s have moved to using 3.4.x of etcd which > allows for 1.5MiB limit). Once etcd is upgraded in all the popular k8s > clusters, then we can hope to overcome this limitation. e.g. > [https://etcd.io/docs/v3.4.0/dev-guide/limit/] version of etcd allows for > higher limit on each entry. > Even if that does not happen, there are other ways to overcome this > limitation, for example, we can have config files split across multiple > configMaps. We need to discuss, and prioritise, this issue takes the > straightforward approach of skipping files that cannot be accommodated within > 1.5MiB limit and WARNING the user about the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org