Re: What is the most efficient and scalable way to get all the recommendation results from ALS model ?

+1 On Mar 21, 2016 09:52, "Hiroyuki Yamada" wrote: > Could anyone give me some advices or recommendations or usual ways to do > this ? > > I am trying to get all (probably top 100) product recommendations for each > user from a model (MatrixFactorizationModel), > but I

Re: This simple UDF is not working!

olumnName", toDate(src(s"$columnName"), lit(s"dd/MM/"))) Regards, Dhaval Modi dhavalmod...@gmail.com On 26 March 2016 at 04:34, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Ted, > > I decided to take a short cut here. I created the map le

Re: Spark SQL drops the HIVE table in "overwrite" mode while writing into table

i, > > is the table that you are trying to overwrite an external table or > temporary table created in hivecontext? > > > Regards, > Gourav Sengupta > > On Sat, Mar 5, 2016 at 3:01 PM, Dhaval Modi <dhavalmod...@gmail.com> > wrote: > >> Hi Team, >> >>

Spark SQL drops the HIVE table in "overwrite" mode while writing into table

ive/warehouse/tgt_table>* at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) ++++++ Regards, Dhaval Modi dhavalmod...@gmail.com

Re: How to set NameSpace while storing from Spark to HBase using saveAsNewAPIHadoopDataSet

Replace with ":" Regards, Dhaval Modi On 19 December 2016 at 13:10, Rabin Banerjee <dev.rabin.baner...@gmail.com> wrote: > HI All, > > I am trying to save data from Spark into HBase using saveHadoopDataSet > API . Please refer the below code . Code is working fine

Re: Advice on multiple streaming job

't any > conflict. > > Le sam. 5 mai 2018 à 17:10, Dhaval Modi <dhavalmod...@gmail.com> a écrit : > >> Hi All, >> >> Need advice on executing multiple streaming jobs. >> >> Problem:- We have 100's of streaming job. Every streaming job uses n

Re: Advice on multiple streaming job

g, but will manage the ports for you ensuring there isn't any >> conflict. >> >> Le sam. 5 mai 2018 à 17:10, Dhaval Modi <dhavalmod...@gmail.com> a >> écrit : >> >>> Hi All, >>> >>> Need advice on executing multiple streaming jobs. >

Re: Advice on multiple streaming job

messages needs to be flattened and stored in Hive. For these 100's of topic, currently we have 100's of jobs running independently and using different UI port. Regards, Dhaval Modi dhavalmod...@gmail.com On 7 May 2018 at 13:53, Gerard Maas <gerard.m...@gmail.com> wrote: > Dhaval,

Advice on multiple streaming job

this situation? or Am I missing any thing? Thanking you in advance. Regards, Dhaval Modi dhavalmod...@gmail.com

How to stop streaming jobs

guide what are the other ways to stop these jobs? Regards, Dhaval Modi dhavalmod...@gmail.com

Re: Properly stop applications or jobs within the application

@sagar - YARN kill is not a reliable process for spark streaming. Regards, Dhaval Modi dhavalmod...@gmail.com On 8 March 2018 at 17:18, bsikander wrote: > I am running in Spark standalone mode. No YARN. > > anyways, yarn application -kill is a manual process. I donot wan

Re: Stopping StreamingContext

+1 Regards, Dhaval Modi dhavalmod...@gmail.com On 29 March 2018 at 19:57, Sidney Feiner wrote: > Hey, > > I have a Spark Streaming application processing some events. > > Sometimes, I want to stop the application if a get a specific event. I > collect the executor's res

Re: Stopping a Spark Streaming Context gracefully

+1 Regards, Dhaval Modi dhavalmod...@gmail.com On 8 November 2017 at 00:06, Bryan Jeffrey wrote: > Hello. > > I am running Spark 2.1, Scala 2.11. We're running several Spark streaming > jobs. In some cases we restart these jobs on an occasional basis. We have > code

Re: Convert a line of String into column

Hi, 1st convert "lines" to dataframe. You will get one column with original string in one row. Post this, use string split on this column to convert to Array of String. After This, you can use explode function to have each element of the array as columns. On Wed 2 Oct, 2019, 03:18 , wrote: