Re: Is there "EXCEPT ALL" in Spark SQL?

To add to it, is there any specific documentation or reference where we could check out what SQL functions and features are available in spark spl for a specific sparksql version.? Thanks,Upendra On Thu, Jul 6, 2017 at 2:22 PM, jeff saremi wrote: I tried this

Adding header to an rdd before saving to text file

I am reading a CSV(file has headers header 1st,header2) and generating rdd,  After few transformations I create an rdd and finally write it to a txt file.  What's the best way to add the header from source file, into rdd and have it available as header into new file I.e, when I transform the rdd

Re: Is there a way to do conditional group by in spark 2.1.1?

Use a function Sent from Yahoo Mail on Android On Sat, Jun 3, 2017 at 5:01 PM, kant kodali wrote: Hi All, Is there a way to do conditional group by in spark 2.1.1? other words, I want to do something like this if (field1 == "foo") {        df.groupBy(field1) } else

Re: Structured Streaming from Parquet

Paul, Did you try, writing to disk rather than in memory. When files are large depending upon which one of quality (performance)/quantity  You want to have, writing to disk would get the load of executors down and will pass to stage where format your data in app2. Other options are to use Kafka

Jupyter spark Scala notebooks

What's the best way to use jupyter with Scala spark. I tried Apache toree and created a kernel but did not get it working. I believe there is a better way. Please suggest any best practices. Sent from Yahoo Mail on Android