Andrew Or created SPARK-5932: -------------------------------- Summary: Use consistent naming for byte properties Key: SPARK-5932 URL: https://issues.apache.org/jira/browse/SPARK-5932 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.0.0 Reporter: Andrew Or Assignee: Andrew Or
This is SPARK-5931's sister issue. The naming of existing byte configs is inconsistent. We currently have the following throughout the code base: spark.reducer.maxMbInFlight (mb) spark.kryoserializer.buffer.mb (mb) spark.shuffle.file.buffer.kb (kb) spark.broadcast.blockSize (kb) spark.executor.logs.rolling.size.maxBytes (bytes) spark.io.compression.snappy.block.size (bytes) Instead, my proposal is to simplify the config name itself and make everything accept time using the following format: 500b, 2k, 100m, 46g, similar to what we currently use for our memory settings. For instance: spark.reducer.maxSizeInFlight = 10m spark.kryoserializer.buffer = 2m spark.shuffle.file.buffer = 10k spark.broadcast.blockSize = 20k spark.executor.logs.rolling.maxSize = 500b spark.io.compression.snappy.blockSize = 200b We should do this soon before we keep introducing more time configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org