Chanced upon spill related config which exhibit same pattern ... - Mridul
On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: > I also just went over the config options to see how pervasive this is. In > addition to speculation, there is one more "conflict" of this kind: > > spark.locality.wait > spark.locality.wait.node > spark.locality.wait.process > spark.locality.wait.rack > > > spark.speculation > spark.speculation.interval > spark.speculation.multiplier > spark.speculation.quantile > > > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia > <matei.zaha...@gmail.com>wrote: > >> This is definitely an important issue to fix. Instead of renaming >> properties, one solution would be to replace Typesafe Config with just >> reading Java system properties, and disable config files for this release. >> I kind of like that over renaming. >> >> Matei >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >> >> > Hi, >> > >> > Speculation was an example, there are others in spark which are >> > affected by this ... >> > Some of them have been around for a while, so will break existing >> code/scripts. >> > >> > Regards, >> > Mridul >> > >> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >> wrote: >> >> change spark.speculation to spark.speculation.switch? >> >> >> >> maybe we can restrict that all properties in Spark should be "three >> levels" >> >> >> >> >> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >> >wrote: >> >> >> >>> Hi, >> >>> >> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >> >>> broken some of the system properties we use in spark. >> >>> >> >>> For example: if we have both >> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >> >>> set, then the spark.speculation property is dropped. >> >>> >> >>> The rules of parseProperty actually document this clearly [1] >> >>> >> >>> >> >>> I am not sure what the right fix here would be (other than replacing >> >>> use of config that is). >> >>> >> >>> Any thoughts ? >> >>> I would vote -1 for 0.9 to be released before this is fixed. >> >>> >> >>> >> >>> Regards, >> >>> Mridul >> >>> >> >>> >> >>> [1] >> >>> >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >> >>> >> >>
