Re: Mulitple joins with same Dataframe throws AnalysisException: resolved attribute(s)

Spark 2.3.0 has this problem upgrade it to 2.3.1 Sent from my iPhone > On Jul 19, 2018, at 2:13 PM, Nirav Patel wrote: > > corrected subject line. It's missing attribute error not ambiguous reference > error. > >> On Thu, Jul 19, 2018 at 2:11 PM, Nirav Patel wrote: >> I am getting

How does databricks ui work ?

Hi All, I had seen some talks where a databricks host writes a sql query in the front end and clicks run then the results are pushed to front end as a stream(the graphs are updated as they get new responses) this pushing data to the front end part I can assume is done via web sockets however

Re: [Spark] Can Apache Spark be used with time series processing?

I don't see why not Sent from my iPhone > On Aug 24, 2017, at 1:52 PM, Alexandr Porunov > wrote: > > Hello, > > I am new in Apache Spark. I need to process different time series data > (numeric values which depend on time) and react on next actions: > 1. Data is

Re: Does Spark SQL uses Calcite?

I also wonder why there isn't a jdbc connector for spark sql? Sent from my iPhone > On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: > > Yes, it's more used in Hive than Spark > > Sent from my iPhone > Pardon the dumb thumb typos :) > >> On Aug 10, 2017, at 2:24 PM,

Re: Is there a Kafka sink for Spark Structured Streaming

Hi! Is this possible possible in spark 2.1.1? Sent from my iPhone > On May 19, 2017, at 5:55 AM, Patrick McGloin > wrote: > > # Write key-value data from a DataFrame to a Kafka topic specified in an > option > query = df \ > .selectExpr("CAST(userId AS STRING)

Is there a Kafka sink for Spark Structured Streaming

Is there a Kafka sink for Spark Structured Streaming ? Sent from my iPhone

Re: Jupyter spark Scala notebooks

Which of these notebooks can help populate real time graphs through web socket or some sort of push mechanism? Sent from my iPhone > On May 17, 2017, at 8:50 PM, Stephen Boesch wrote: > > Jupyter with toree works well for my team. Jupyter is well more refined vs >

Re: spark intermediate data fills up the disk

Hi! Yes these files are for shuffle blocks however they need to be cleaned as well right? I had been running a streaming application for 2 days. The third day my disk fills up with all .index and .data files and my assumption is that these files had been there since the start of my streaming