Oh great, just saw the PR from Matei ... for some odd reason, the dev mails are coming to be horribly delayed.
Thanks, Mridul On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Hey Mridul this was patched and we cut a new release candidate. There > were several different config options which had a.b and a.b.c... they > should all work in the new RC. > > On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mri...@gmail.com> wrote: >> Chanced upon spill related config which exhibit same pattern ... >> >> - Mridul >> >> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: >>> I also just went over the config options to see how pervasive this is. In >>> addition to speculation, there is one more "conflict" of this kind: >>> >>> spark.locality.wait >>> spark.locality.wait.node >>> spark.locality.wait.process >>> spark.locality.wait.rack >>> >>> >>> spark.speculation >>> spark.speculation.interval >>> spark.speculation.multiplier >>> spark.speculation.quantile >>> >>> >>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >>> <matei.zaha...@gmail.com>wrote: >>> >>>> This is definitely an important issue to fix. Instead of renaming >>>> properties, one solution would be to replace Typesafe Config with just >>>> reading Java system properties, and disable config files for this release. >>>> I kind of like that over renaming. >>>> >>>> Matei >>>> >>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >>>> wrote: >>>> >>>> > Hi, >>>> > >>>> > Speculation was an example, there are others in spark which are >>>> > affected by this ... >>>> > Some of them have been around for a while, so will break existing >>>> code/scripts. >>>> > >>>> > Regards, >>>> > Mridul >>>> > >>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >>>> wrote: >>>> >> change spark.speculation to spark.speculation.switch? >>>> >> >>>> >> maybe we can restrict that all properties in Spark should be "three >>>> levels" >>>> >> >>>> >> >>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >>>> >wrote: >>>> >> >>>> >>> Hi, >>>> >>> >>>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>>> >>> broken some of the system properties we use in spark. >>>> >>> >>>> >>> For example: if we have both >>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>> >>> set, then the spark.speculation property is dropped. >>>> >>> >>>> >>> The rules of parseProperty actually document this clearly [1] >>>> >>> >>>> >>> >>>> >>> I am not sure what the right fix here would be (other than replacing >>>> >>> use of config that is). >>>> >>> >>>> >>> Any thoughts ? >>>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>>> >>> >>>> >>> >>>> >>> Regards, >>>> >>> Mridul >>>> >>> >>>> >>> >>>> >>> [1] >>>> >>> >>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>> >>> >>>> >>>>
