Hi,
UpdateStateByKey : if you can brief the issue you are facing with
this,that will be great.
Regarding not keeping whole dataset in memory, you can tweak the parameter
of remember, such that it does checkpoint at appropriate time.
Thanks
Twinkle
On Thursday, June 18, 2015, Nipun Arora
Thanks Saisai.
On Wed, May 20, 2015 at 11:23 AM, Saisai Shao
wrote:
> I think here is the PR https://github.com/apache/spark/pull/2994 you
> could refer to.
>
> 2015-05-20 13:41 GMT+08:00 twinkle sachdeva :
>
>> Hi,
>>
>> As Spark streaming is being nicely i
Hi,
As Spark streaming is being nicely integrated with consuming messages from
Kafka, so I thought of asking the forum, that is there any implementation
available for pushing data to Kafka from Spark Streaming too?
Any link(s) will be helpful.
Thanks and Regards,
Twinkle
Hi,
Can you please share your compression etc settings, which you are using.
Thanks,
Twinkle
On Wed, May 6, 2015 at 4:15 PM, Jianshi Huang
wrote:
> I'm facing this error in Spark 1.3.1
>
> https://issues.apache.org/jira/browse/SPARK-4105
>
> Anyone knows what's t
somebody please comment if it is a bug or some intended behaviour w.r.t
performance or some other bottleneck.
--Twinkle
On Mon, Apr 20, 2015 at 2:47 PM, Archit Thakur
wrote:
> Hi Twinkle,
>
> We have a use case in where we want to debug the reason of how n why an
> executor got killed
Hi Archit,
What is your use case and what kind of metrics are you planning to add?
Thanks,
Twinkle
On Fri, Apr 17, 2015 at 4:07 PM, Archit Thakur
wrote:
> Hi,
>
> We are planning to add new Metrics in Spark for the executors that got
> killed during the execution. Was just curio
Hi,
In one of the application we have made which had no clone stuff, we have
set the value of spark.storage.memoryFraction to very low, and yes that
gave us performance benefits.
Regarding that issue, you should also look at the data you are trying to
broadcast, as sometimes creating that data st
Hi,
If you have the same spark context, then you can cache the query result via
caching the table ( sqlContext.cacheTable("tableName") ).
Maybe you can have a look at OOyola server also.
On Tue, Apr 14, 2015 at 11:36 AM, Akhil Das
wrote:
> You can use a tachyon based storage for that and eve
of this setting? ( which again let me think over this setting ).
Comments please.
Thanks,
Twinkle
to a window
duration.
I will upload the PR shortly.
Thanks,
Twinkle
On Tue, Apr 7, 2015 at 2:02 AM, Sandy Ryza wrote:
> What's the advantage of killing an application for lack of resources?
>
> I think the rationale behind killing an app based on executor failures is
> that, if we
f time, then we should fail the application.
Adding time factor here, will allow some window for spark to get more
executors allocated if some of them fails.
Thoughts please.
Thanks,
Twinkle
On Wed, Apr 1, 2015 at 10:19 PM, Sandy Ryza wrote:
> That's a good question, Twinkle.
>
> O
t at some point even a single executor
failure ( which application could have survived ) can make the application
quit.
Sending it to the community to listen what kind of behaviour / strategy
they think will be suitable for long running spark jobs or spark streaming
jobs.
Thanks and Regards,
Twinkle
ed to the
> spark cluster based on the priority. jobs will lower priority or less than
> some threshold will be discarded.
>
> Thanks,
> Abhi
>
>
> On Mon, Mar 16, 2015 at 10:36 PM, twinkle sachdeva <
> twinkle.sachd...@gmail.com> wrote:
>
>> Hi Abhi,
&
>> Thanks,
>> Abhi
>>
>>
>> On Mon, Mar 16, 2015 at 9:48 PM, twinkle sachdeva <
>> twinkle.sachd...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Maybe this is what you are looking for :
>>> http://spark.apache.org/docs/1.2.0
Hi,
Maybe this is what you are looking for :
http://spark.apache.org/docs/1.2.0/job-scheduling.html#fair-scheduler-pools
Thanks,
On Mon, Mar 16, 2015 at 8:15 PM, abhi wrote:
> Hi
> Current all the jobs in spark gets submitted using queue . i have a
> requirement where submitted job will genera
disassociated*
How can i make this to happen faster?
Thanks,
Twinkle
wrote:
> Mostly, that particular executor is stuck on GC Pause, what operation are
> you performing? You can try increasing the parallelism if you see only 1
> executor is doing the task.
>
> Thanks
> Best Regards
>
> On Fri, Feb 27, 2015 at 11:39 AM, twinkle sachdeva <
&
nager BlockManagerId(7, TMO-DN73, 34106) with no recent heart beats:
80515ms exceeds 45000ms
I am using spark 1.2.1.
Any pointer(s) ?
Thanks,
Twinkle
Hi,
What is the file format which is used to write files while shuffle write?
Is it dependent on the spark shuffle manager or output format?
Is it possible to change the file format for shuffle, irrespective of the
output format of the file?
Thanks,
Twinkle
; the Hadoop Configuration to suggest a max/min split size, and
> therefore bound the number of partitions you get back.
>
> On Thu, Feb 19, 2015 at 11:07 AM, twinkle sachdeva
> wrote:
> > Hi,
> >
> > In our job, we need to process the data in small chunks, so as to avoid
from older API?
I am little bit aware of split size stuff, but not much aware regarding any
promise that minimum number of partitions criteria gets satisfied or not.
Any pointers will be of help.
Thanks,
Twinkle
Hi,
Try running following in the spark folder:
bin/*run-example *SparkPi 10
If this runs fine, just see the set of arguments being passed via this
script, and try in similar way.
Thanks,
On Thu, Oct 16, 2014 at 2:59 PM, Christophe Préaud <
christophe.pre...@kelkoo.com> wrote:
> Hi,
>
> I ha
Hi,
I have been using spark sql with yarn.
It works fine with yarn-client mode, but with yarn-cluster mode, we are
facing 2 issues. Is yarn-cluster mode not recommended for spark-sql using
hiveContext ??
*Problem #1*
We are not able to use any query with very simple filtering operation
"like",
Hi,
Can somebody please share the plans regarding java version's support for
apache spark 1.2.0 or near future releases.
Will java 8 become the all feature supported version in apache spark 1.2 or
java 1.7 will suffice?
Thanks,
, 2014 at 5:13 PM, Cheng Lian wrote:
> H Twinkle,
>
> The failure is caused by case sensitivity. The temp table actually stores
> the original un-analyzed logical plan, thus field names remain capital (F1,
> F2, etc.). I believe this issue has already been fixed by PR #2382
> &
Hi,
I am using Hive Context to fire the sql queries inside spark. I have
created a schemaRDD( Let's call it cachedSchema ) inside my code.
If i fire a sql query ( Query 1 ) on top of it, then it works.
But if I refer to Query1's result inside another sql, that fails. Note that
I have already regi
Hi,
Has anyone else also experienced
https://issues.apache.org/jira/browse/SPARK-2604?
It is an edge case scenario of mis configuration, where the executor memory
asked is same as the maximum allowed memory by yarn. In such situation,
application stays in hang state, and the reason is not logged
28 matches
Mail list logo