Re: What do I loose if I run spark without using HDFS or Zookeeper?

I understand now that for I cannot use spark streaming window operation without checkpointing to HDFS as pointed out by @Ofir but Without window operation I don't think we can do much with spark streaming. so since it is very essential can I use Cassandra as a distributed storage? If so, can I see

Write parquet file from Spark Streaming

Hi Everyone, Does anyone know how to write parquet file after parsing data in Spark Streaming? Thanks, Kevin.

Re: Reading parquet files into Spark Streaming

Forgot to paste the link... http://ramblings.azurewebsites.net/2016/01/26/save-parquet-rdds-in-apache-spark/ On Sat, 27 Aug 2016, 19:18 Sebastian Piu, wrote: > Hi Renato, > > Check here on how to do it, it is in Java but you can translate it to > Scala if that is what

Re: Reading parquet files into Spark Streaming

Hi Renato, Check here on how to do it, it is in Java but you can translate it to Scala if that is what you need. Cheers On Sat, 27 Aug 2016, 14:24 Renato Marroquín Mogrovejo, < renatoj.marroq...@gmail.com> wrote: > Hi Akhilesh, > > Thanks for your response. > I am using Spark 1.6.1 and what I

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Nguyen for the link. I installed Super Refresh as ADD on to Chrome. By default the refresh is stop until you set it to x seconds. However, the issue we have is that Spark UI comes with 6+ tabs and you have to repeat the process for each tab. However, that messes up the things. For example

Re: Please assist: Building Docker image containing spark 2.0

all good. Tal's suggestion did it. i shud have read the manual first :( tx for assistance On Sat, Aug 27, 2016 at 9:06 AM, Marco Mistroni wrote: > Thanks, i'll follow advice and try again > > kr > marco > > On Sat, Aug 27, 2016 at 4:04 AM, Mike Metzger

Re: Is there anyway Spark UI is set to poll and refreshes itself

The simplest solution that I found: using an browser extension which do that for you :D. For example, if you are using Chrome, you can use this extension: https://chrome.google.com/webstore/detail/easy-auto-refresh/aabcgdmkeabbnleenpncegpcngjpnjkc/related?hl=en An other way, but a bit more

Re: Reading parquet files into Spark Streaming

Hi Akhilesh, Thanks for your response. I am using Spark 1.6.1 and what I am trying to do is to ingest parquet files into the Spark Streaming, not in batch operations. val ssc = new StreamingContext(sc, Seconds(5)) ssc.sparkContext.hadoopConfiguration.set("parquet.read.support.class",

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi All, GitHub project SparkUIDashboard created here [image: Inline images 2] Let use put the show on the road :) Cheers Dr Mich Talebzadeh LinkedIn *

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi, There's no better way to start a project than...github it :-) Create a new project, clone it and do dzieła! (= go ahead in Polish). Jacek On 27 Aug 2016 10:42 a.m., "Sivakumaran S" wrote: > I would love to participate in developing a dashboard of some sort in lieu

Issues with Spark On Hbase Connector and versions

Regarding hbase connector by hortonworks https://github.com/hortonworks-spark/shc, it would be great if someone can answer these 1. What versions of Hbase & Spark expected? I could not run examples provided using spark 1.6.0 & hbase 1.2.0 2. I get error when i run example provided here

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Sivakumaran I don't think we can use Zeppelin for this purpose. It is not a real time dashboard or can it be. I use it but much like Tableau with added Scala programming. Does anyone know of open source real time dashboards? Cheers Dr Mich Talebzadeh LinkedIn *

Re: Is there anyway Spark UI is set to poll and refreshes itself

Are we actually looking for a eal time dashboard of some sort for Spark UI interface? After all one can think a real time dashboard can do this! HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Please assist: Building Docker image containing spark 2.0

Thanks, i'll follow advice and try again kr marco On Sat, Aug 27, 2016 at 4:04 AM, Mike Metzger wrote: > I would also suggest building the container manually first and setup > everything you specifically need. Once done, you can then grab the history > file, pull

Re: Reading parquet files into Spark Streaming

Hi Renato, Which version of Spark are you using? If spark version is 1.3.0 or more then you can use SqlContext to read the parquet file which will give you DataFrame. Please follow the below link: https://spark.apache.org/docs/1.5.0/sql-programming-guide.html#loading-data-programmatically