Hi, Speculation was an example, there are others in spark which are affected by this ... Some of them have been around for a while, so will break existing code/scripts.
Regards, Mridul On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > change spark.speculation to spark.speculation.switch? > > maybe we can restrict that all properties in Spark should be "three levels" > > > On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com>wrote: > >> Hi, >> >> Unless I am mistaken, the change to using typesafe ConfigFactory has >> broken some of the system properties we use in spark. >> >> For example: if we have both >> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >> set, then the spark.speculation property is dropped. >> >> The rules of parseProperty actually document this clearly [1] >> >> >> I am not sure what the right fix here would be (other than replacing >> use of config that is). >> >> Any thoughts ? >> I would vote -1 for 0.9 to be released before this is fixed. >> >> >> Regards, >> Mridul >> >> >> [1] >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>
