of implementation possible with spark streaming ?
If not then do I need to use some other tool along with spark streaming to
include this kind of processing ? What are the possible options for me?
Thanks in advance.
Thanks,
Udbhav Agarwal
p 2016 23:29, "Udbhav Agarwal"
<udbhav.agar...@syncoms.com<mailto:udbhav.agar...@syncoms.com>> wrote:
Thank you Ayan for a reply.
Source is kafka but I am reading from this source in my main stream. I will
perform some operations here. Then I want to send the output of th
?
Thanks,
Udbhav
From: ayan guha [mailto:guha.a...@gmail.com]
Sent: Thursday, September 15, 2016 6:43 PM
To: Udbhav Agarwal <udbhav.agar...@syncoms.com>
Cc: user <user@spark.apache.org>
Subject: Re: Spark processing Multiple Streams from a single stream
Depending on source. For exampl
stream and are
always active and waiting for input. Can I implement such a process with spark
streaming ? How ?
Thanks in advance.
Thanks,
Udbhav Agarwal
if we want to analyse
related/connected information e.g. friends of friends and other stuffs.
I really appreciate your valuable help Robin. Thank you In advance.
Udbhav.
From: Robin East [mailto:robin.e...@xense.co.uk]
Sent: Thursday, February 25, 2016 7:42 PM
To: Udbhav Agarwal <udbhav.a
as to how I can keep this graph always available
to user even if any background process is happening over it.
Thanks,
Udbhav Agarwal
on. This is a case when I want to
add vertices to graph as and when they are coming in our system since it’s a
real time system which I am trying to build so vertices will be keep on coming.
Thanks.
From: Robin East [mailto:robin.e...@xense.co.uk]
Sent: Wednesday, February 24, 2016 3:54 PM
To: Udbhav Agarwal
ces, gEdges, defaultUser)
inputGraph.cache()
gVertices = inputGraph.vertices
gVertices.cache()
val count = gVertices.count
println(count);
return 1;
}
From: Robin East [mailto:robin.e...@xense.co.uk]
Sent: Tuesday, February 23, 2016 8:15 PM
To: Udbhav Agarwal <udbha
am missing with the syntax as I have seen the API
documentation where only vertices.reindex() is mentioned.
Thanks,
Udbhav Agarwal
Hi ,
I am adding bunch of vertices in a graph in graphx using the following method .
I am facing the problem of latency. First time an addition of say 400 vertices
to a graph with 100,000 nodes takes around 7 seconds. next time its taking 15
seconds. So every subsequent adds are taking more
is not picking the updated data from the
table. Its showing the old data. Can somebody tell how to let spark know I have
updated my database after spark has created Rdds.
Thanks,
Udbhav Agarwal
Thanks Akhil,
What more info should I give so we can estimate query time in my scenario?
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 13 March, 2015 12:01 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org
Subject: Re: spark sql performance
That totally depends
Lets say am using 4 machines with 3gb ram. My data is customers records with 5
columns each in two tables with 0.5 million records. I want to perform join
query on these two tables.
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 13 March, 2015 12:16 PM
Okay Akhil! Thanks for the information.
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 13 March, 2015 12:34 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org
Subject: Re: spark sql performance
Can't say that unless you try it.
Thanks
Best Regards
On Fri, Mar 13
Sounds great!
So can I expect response time in milliseconds from the join query over this
much data ( 0.5 million in each table) ?
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 13 March, 2015 12:27 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org
Subject: Re
Additionally I wanted to tell that presently I was running the query on one
machine with 3gm ram and the join query was taking around 6 seconds.
Thanks,
Udbhav Agarwal
From: Udbhav Agarwal
Sent: 13 March, 2015 12:45 PM
To: 'Akhil Das'
Cc: user@spark.apache.org
Subject: RE: spark sql performance
Okay Akhil.
I am having 4 cores cpu.(2.4 ghz)
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 13 March, 2015 1:07 PM
To: Udbhav Agarwal
Cc: user@spark.apache.org
Subject: Re: spark sql performance
You can see where it is spending time, whether there is any GC
Thanks Akhil.
Additionaly if we want to do sql query we need to create JavaPairRdd, then
JavaRdd, then JavaSchemaRdd and then sqlContext.sql(sql query). Ryt ?
Thanks,
Udbhav Agarwal
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: 12 March, 2015 11:43 AM
To: Udbhav Agarwal
Cc: user
Thanks Todd,
But this link is also based on scala, I was looking for some help with java
Apis.
Thanks,
Udbhav Agarwal
From: Todd Nist [mailto:tsind...@gmail.com]
Sent: 12 March, 2015 5:28 PM
To: Udbhav Agarwal
Cc: Akhil Das; user@spark.apache.org
Subject: Re: hbase sql query
Have you
Hi,
What is query time for join query on hbase with spark sql. Say tables in hbase
have 0.5 million records each. I am expecting a query time (latency) in
milliseconds with spark sql. Can this be possible ?
Thanks,
Udbhav Agarwal
JavaSchemaRdds and sqlContext.sql only. Isn’t it ??
Thanks,
Udbhav Agarwal
From: Todd Nist [mailto:tsind...@gmail.com]
Sent: 12 March, 2015 6:19 PM
To: Udbhav Agarwal
Cc: Akhil Das; user@spark.apache.org
Subject: Re: hbase sql query
Ah, missed that java was a requirement. What distribution
Hi,
How can we simply cache hbase table and do sql query via java api in spark.
Thanks,
Udbhav Agarwal
22 matches
Mail list logo
