Andrew Or created SPARK-5932:
--------------------------------
Summary: Use consistent naming for byte properties
Key: SPARK-5932
URL: https://issues.apache.org/jira/browse/SPARK-5932
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Andrew Or
Assignee: Andrew Or
This is SPARK-5931's sister issue.
The naming of existing byte configs is inconsistent. We currently have the
following throughout the code base:
spark.reducer.maxMbInFlight (mb)
spark.kryoserializer.buffer.mb (mb)
spark.shuffle.file.buffer.kb (kb)
spark.broadcast.blockSize (kb)
spark.executor.logs.rolling.size.maxBytes (bytes)
spark.io.compression.snappy.block.size (bytes)
Instead, my proposal is to simplify the config name itself and make everything
accept time using the following format: 500b, 2k, 100m, 46g, similar to what we
currently use for our memory settings. For instance:
spark.reducer.maxSizeInFlight = 10m
spark.kryoserializer.buffer = 2m
spark.shuffle.file.buffer = 10k
spark.broadcast.blockSize = 20k
spark.executor.logs.rolling.maxSize = 500b
spark.io.compression.snappy.blockSize = 200b
We should do this soon before we keep introducing more time configs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
