Re: What is correct behavior for spark.task.maxFailures?

2017-04-24 Thread Dongjin Lee
Sumit, I think the post below is describing the very case of you. https://blog.cloudera.com/blog/2017/04/blacklisting-in-apache-spark/ Regards, Dongjin -- Dongjin Lee Software developer in Line+. So interested in massive-scale machine learning. facebook: http://www.facebook.com

Re: Issues: Generate JSON with null values in Spark 2.0.x

2017-03-21 Thread Dongjin Lee
, 2017 at 4:44 PM, Chetan Khatri <chetan.opensou...@gmail.com> wrote: > Exactly. > > On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee <dong...@apache.org> wrote: > >> Hello Chetan, >> >> Could you post some code? If I understood correctly, you are trying to &

Re: Issues: Generate JSON with null values in Spark 2.0.x

2017-03-11 Thread Dongjin Lee
to SparkSQL[Row] but in scala how can > we generate json will null values as a Dataframe ? > > Thanks. > -- *Dongjin Lee* *Software developer in Line+.So interested in massive-scale machine learning.facebook: www.facebook.com/dongjin.lee.kr <http://www.facebook.com/dongjin.lee.kr&