Mridul Muralidharan created SPARK-6050:
--
Summary: Spark on YARN does not work --executor-cores is specified
Key: SPARK-6050
URL: https://issues.apache.org/jira/browse/SPARK-6050
Project: Spark
to LargeByteBuffer, seems
> promising.
>
> thanks,
> Imran
>
> On Tue, Feb 3, 2015 at 7:32 PM, Mridul Muralidharan > wrote:
>
>> That is fairly out of date (we used to run some of our jobs on it ... But
>> that is forked off 1.1 actually).
>>
>> Regard
to LargeByteBuffer, seems
> promising.
>
> thanks,
> Imran
>
> On Tue, Feb 3, 2015 at 7:32 PM, Mridul Muralidharan > wrote:
>
>> That is fairly out of date (we used to run some of our jobs on it ... But
>> that is forked off 1.1 actually).
>>
>> Regard
That is fairly out of date (we used to run some of our jobs on it ... But
that is forked off 1.1 actually).
Regards
Mridul
On Tuesday, February 3, 2015, Imran Rashid wrote:
> Thanks for the explanations, makes sense. For the record looks like this
> was worked on a while back (and maybe the wo
That is fairly out of date (we used to run some of our jobs on it ... But
that is forked off 1.1 actually).
Regards
Mridul
On Tuesday, February 3, 2015, Imran Rashid wrote:
> Thanks for the explanations, makes sense. For the record looks like this
> was worked on a while back (and maybe the wo
Congratulations !
Keep up the good work :-)
Regards
Mridul
On Tuesday, February 3, 2015, Matei Zaharia wrote:
> Hi all,
>
> The PMC recently voted to add three new committers: Cheng Lian, Joseph
> Bradley and Sean Owen. All three have been major contributors to Spark in
> the past year: Cheng
I second that !
Would also be great if the JIRA was updated accordingly too.
Regards,
Mridul
On Wed, Dec 3, 2014 at 1:53 AM, Kay Ousterhout wrote:
> Hi all,
>
> I've noticed a bunch of times lately where a pull request changes to be
> pretty different from the original pull request, and the tit
CAL machine. Of
> course, this would add a bunch of complexity to the TSM, hence the earlier
> decision that the added complexity may not be worth it.
>
> -Kay
>
> On Thu, Nov 13, 2014 at 12:11 PM, Mridul Muralidharan
> wrote:
>>
>> Instead of setting spark.loc
Instead of setting spark.locality.wait, try setting individual
locality waits specifically.
Namely, spark.locality.wait.PROCESS_LOCAL to high value (so that
process local tasks are always scheduled in case the task set has
process local tasks).
Set spark.locality.wait.NODE_LOCAL and spark.locality
[
https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178162#comment-14178162
]
Mridul Muralidharan commented on SPARK-4030:
We have also needed to
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173489#comment-14173489
]
Mridul Muralidharan edited comment on SPARK-3948 at 10/16/14 7:3
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173489#comment-14173489
]
Mridul Muralidharan commented on SPARK-3948:
Damn, this sucks :
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173413#comment-14173413
]
Mridul Muralidharan commented on SPARK-3948:
Not exactly, what I
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172585#comment-14172585
]
Mridul Muralidharan commented on SPARK-3948:
[~jerryshao] great work
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172226#comment-14172226
]
Mridul Muralidharan commented on SPARK-3948:
Note, "t" is just
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172225#comment-14172225
]
Mridul Muralidharan commented on SPARK-3948:
That is weird, I tri
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172134#comment-14172134
]
Mridul Muralidharan commented on SPARK-3948:
[~jerryshao] Just to cla
[
https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172116#comment-14172116
]
Mridul Muralidharan commented on SPARK-3948:
[~joshrosen] Assuming there
[
https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167303#comment-14167303
]
Mridul Muralidharan commented on SPARK-3889:
The status says fixed - what
Brilliant stuff ! Congrats all :-)
This is indeed really heartening news !
Regards,
Mridul
On Fri, Oct 10, 2014 at 8:24 PM, Matei Zaharia wrote:
> Hi folks,
>
> I interrupt your regularly scheduled user / dev list to bring you some pretty
> cool news for the project, which is that we've been a
Brilliant stuff ! Congrats all :-)
This is indeed really heartening news !
Regards,
Mridul
On Fri, Oct 10, 2014 at 8:24 PM, Matei Zaharia wrote:
> Hi folks,
>
> I interrupt your regularly scheduled user / dev list to bring you some pretty
> cool news for the project, which is that we've been a
[
https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164775#comment-14164775
]
Mridul Muralidharan commented on SPARK-3847:
[~joshrosen] array hash
[
https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164095#comment-14164095
]
Mridul Muralidharan commented on SPARK-3561:
I agree with [~pwendell]
[
https://issues.apache.org/jira/browse/SPARK-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163407#comment-14163407
]
Mridul Muralidharan commented on SPARK-3847:
Wow, nice bug ! Thi
[
https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163376#comment-14163376
]
Mridul Muralidharan commented on SPARK-3561:
[~ozhurakousky] I think
[
https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163351#comment-14163351
]
Mridul Muralidharan commented on SPARK-3561:
[~pwendell] If I understood
[
https://issues.apache.org/jira/browse/SPARK-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163326#comment-14163326
]
Mridul Muralidharan commented on SPARK-3785:
[~sowen] We had prototyp
[
https://issues.apache.org/jira/browse/SPARK-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151486#comment-14151486
]
Mridul Muralidharan commented on SPARK-3714:
Most of the drawbacks menti
[
https://issues.apache.org/jira/browse/SPARK-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151211#comment-14151211
]
Mridul Muralidharan commented on SPARK-3714:
Have you tried using oozie
[
https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125027#comment-14125027
]
Mridul Muralidharan commented on SPARK-1956:
The recent change
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119385#comment-14119385
]
Mridul Muralidharan commented on SPARK-1476:
WIP version pushed to h
[
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119365#comment-14119365
]
Mridul Muralidharan commented on SPARK-3019:
I will try to push the ver
[
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119362#comment-14119362
]
Mridul Muralidharan commented on SPARK-3019:
Just went over the proposa
and we'll patch it
> and spin a new RC. We can also update the test coverage to cover LZ4.
>
> - Patrick
>
> On Thu, Aug 28, 2014 at 9:27 AM, Mridul Muralidharan
> wrote:
> > Is SPARK-3277 applicable to 1.1 ?
> > If yes, until it is fixed, I am -1 on the releas
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114484#comment-14114484
]
Mridul Muralidharan commented on SPARK-3277:
Sounds great, thx !
I suspec
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114026#comment-14114026
]
Mridul Muralidharan commented on SPARK-3277:
[~hzw] did you notice
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114022#comment-14114022
]
Mridul Muralidharan edited comment on SPARK-3277 at 8/28/14 5:3
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-3277:
---
Attachment: test_lz4_bug.patch
Against master, though I noticed similar changes in
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114014#comment-14114014
]
Mridul Muralidharan commented on SPARK-3277:
[~matei] Attaching a patch w
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-3277:
---
Priority: Blocker (was: Major)
> LZ4 compression cause the the ExternalS
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-3277:
---
Affects Version/s: 1.2.0
1.1.0
> LZ4 compression cause
Is SPARK-3277 applicable to 1.1 ?
If yes, until it is fixed, I am -1 on the release (I am on break, so can't
verify or help fix, sorry).
Regards
Mridul
On 28-Aug-2014 9:33 pm, "Patrick Wendell" wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.1.0!
>
> The ta
[
https://issues.apache.org/jira/browse/SPARK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113741#comment-14113741
]
Mridul Muralidharan commented on SPARK-3277:
This looks like unrel
[
https://issues.apache.org/jira/browse/SPARK-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14107923#comment-14107923
]
Mridul Muralidharan commented on SPARK-3175:
Please add more informatio
Weird that Patrick did not face this while creating the RC.
Essentially the yarn alpha pom.xml has not been updated properly in
the 1.1 branch.
Just change version to '1.1.1-SNAPSHOT' for yarn/alpha/pom.xml (to
make it same as any other pom).
Regards,
Mridul
On Thu, Aug 21, 2014 at 5:09 AM, Ch
[
https://issues.apache.org/jira/browse/SPARK-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102142#comment-14102142
]
Mridul Muralidharan commented on SPARK-3115:
I had a tab open with pr
[
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100043#comment-14100043
]
Mridul Muralidharan commented on SPARK-3019:
Unfortunately, I never went
[
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099910#comment-14099910
]
Mridul Muralidharan commented on SPARK-3019:
Btw, can we do something a
[
https://issues.apache.org/jira/browse/SPARK-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099909#comment-14099909
]
Mridul Muralidharan commented on SPARK-3019:
I am yet to go through
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099115#comment-14099115
]
Mridul Muralidharan commented on SPARK-2089:
For a general case,
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099072#comment-14099072
]
Mridul Muralidharan commented on SPARK-1476:
Based on discussions we had
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095247#comment-14095247
]
Mridul Muralidharan commented on SPARK-2089:
Since I am not maintaining
[
https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092807#comment-14092807
]
Mridul Muralidharan commented on SPARK-2962:
On further investigation
[
https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092427#comment-14092427
]
Mridul Muralidharan edited comment on SPARK-2962 at 8/11/14 4:3
[
https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092431#comment-14092431
]
Mridul Muralidharan commented on SPARK-2962:
Note, I dont think this
[
https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092430#comment-14092430
]
Mridul Muralidharan commented on SPARK-2962:
Hi [~matei],
I am referen
[
https://issues.apache.org/jira/browse/SPARK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092427#comment-14092427
]
Mridul Muralidharan commented on SPARK-2962:
To give more context;
a)
Mridul Muralidharan created SPARK-2962:
--
Summary: Suboptimal scheduling in spark
Key: SPARK-2962
URL: https://issues.apache.org/jira/browse/SPARK-2962
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091881#comment-14091881
]
Mridul Muralidharan commented on SPARK-2931:
[~joshrosen] [~kayouster
[
https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-2931:
---
Attachment: test.patch
A patch to showcase the exception
> getAllowedLocalityLe
[
https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091849#comment-14091849
]
Mridul Muralidharan commented on SPARK-2931:
Checking more, it might
Issue with supporting this imo is the fact that scala-test uses the
same vm for all the tests (surefire plugin supports fork, but
scala-test ignores it iirc).
So different tests would initialize different spark context, and can
potentially step on each others toes.
Regards,
Mridul
On Fri, Aug 8,
[
https://issues.apache.org/jira/browse/SPARK-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091746#comment-14091746
]
Mridul Muralidharan commented on SPARK-2931:
[~kayousterhout] this is w
[
https://issues.apache.org/jira/browse/SPARK-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088018#comment-14088018
]
Mridul Muralidharan edited comment on SPARK-2881 at 8/6/14 6:4
[
https://issues.apache.org/jira/browse/SPARK-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088018#comment-14088018
]
Mridul Muralidharan commented on SPARK-2881:
To add, this will affect s
Just came across this mail, thanks for initiating this discussion Kay.
To add; another issue which recurs is very rapid commit's: before most
contributors have had a chance to even look at the changes proposed.
There is not much prior discussion on the jira or pr, and the time
between submitting th
[
https://issues.apache.org/jira/browse/SPARK-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074186#comment-14074186
]
Mridul Muralidharan commented on SPARK-2685:
We moved to u
Mridul Muralidharan created SPARK-2532:
--
Summary: Fix issues with consolidated shuffle
Key: SPARK-2532
URL: https://issues.apache.org/jira/browse/SPARK-2532
Project: Spark
Issue Type
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061094#comment-14061094
]
Mridul Muralidharan commented on SPARK-2468:
Ah, small files - those
We tried with lower block size for lzf, but it barfed all over the place.
Snappy was the way to go for our jobs.
Regards,
Mridul
On Mon, Jul 14, 2014 at 12:31 PM, Reynold Xin wrote:
> Hi Spark devs,
>
> I was looking into the memory usage of shuffle and one annoying thing is
> the default comp
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14060545#comment-14060545
]
Mridul Muralidharan commented on SPARK-2468:
Writing mmap'ed bu
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14060543#comment-14060543
]
Mridul Muralidharan commented on SPARK-2468:
We map the file content
[
https://issues.apache.org/jira/browse/SPARK-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14060113#comment-14060113
]
Mridul Muralidharan commented on SPARK-2398:
As discussed in the PR,
You are lucky :-) for some of our jobs, in a 8gb container, overhead is
1.8gb !
On 13-Jul-2014 2:40 pm, "nishkamravi2" wrote:
> Github user nishkamravi2 commented on the pull request:
>
> https://github.com/apache/spark/pull/1391#issuecomment-48835560
>
> Sean, the memory_overhead is fair
+1 on advanced mode !
Regards.
Mridul
On Thu, Jul 10, 2014 at 12:55 AM, Reynold Xin wrote:
> Maybe it's time to create an advanced mode in the ui.
>
>
> On Wed, Jul 9, 2014 at 12:23 PM, Kay Ousterhout
> wrote:
>
>> Hi all,
>>
>> I've been doing a bunch of performance measurement of Spark and, a
Hi,
I noticed today that gmail has been marking most of the mails from
spark github/jira I was receiving to spam folder; and I was assuming
it was lull in activity due to spark summit for past few weeks !
In case I have commented on specific PR/JIRA issues and not followed
up, apologies for th
You are ignoring serde costs :-)
- Mridul
On Tue, Jul 8, 2014 at 8:48 PM, Aaron Davidson wrote:
> Tachyon should only be marginally less performant than memory_only, because
> we mmap the data from Tachyon's ramdisk. We do not have to, say, transfer
> the data over a pipe from Tachyon; we can di
[
https://issues.apache.org/jira/browse/SPARK-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054162#comment-14054162
]
Mridul Muralidharan commented on SPARK-2390:
Here, and a bunch of o
[
https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052679#comment-14052679
]
Mridul Muralidharan commented on SPARK-2017:
Sounds great, ability to ge
size = 0 using a compressed bitmap. That way we can still avoid
> requests for zero-sized blocks.
>
>
>
> On Thu, Jul 3, 2014 at 3:12 PM, Reynold Xin wrote:
>
>> Yes, that number is likely == 0 in any real workload ...
>>
>>
>> On Thu, Jul 3, 2014 at 8:01 AM, Mr
[
https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052289#comment-14052289
]
Mridul Muralidharan commented on SPARK-2017:
With aggregated metrics
[
https://issues.apache.org/jira/browse/SPARK-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052275#comment-14052275
]
Mridul Muralidharan commented on SPARK-2277:
Hmm, good point - that PR
[
https://issues.apache.org/jira/browse/SPARK-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-2353:
---
Description:
I suspect the recent changes from SPARK-1937 to compute valid locality
[
https://issues.apache.org/jira/browse/SPARK-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051575#comment-14051575
]
Mridul Muralidharan commented on SPARK-2277:
I have not rechecked that
On Thu, Jul 3, 2014 at 11:32 AM, Reynold Xin wrote:
> On Wed, Jul 2, 2014 at 3:44 AM, Mridul Muralidharan
> wrote:
>
>>
>> >
>> > The other thing we do need is the location of blocks. This is actually
>> just
>> > O(n) because we just ne
Mridul Muralidharan created SPARK-2353:
--
Summary: ArrayIndexOutOfBoundsException in scheduler
Key: SPARK-2353
URL: https://issues.apache.org/jira/browse/SPARK-2353
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050886#comment-14050886
]
Mridul Muralidharan commented on SPARK-2277:
I am not sure I follow
r a reducer (and lack
of ability to throttle).
Regards,
Mridul
>
>
> On Tue, Jul 1, 2014 at 2:51 AM, Mridul Muralidharan
> wrote:
>
>> We had considered both approaches (if I understood the suggestions right) :
>> a) Pulling only map output states for tasks which run on t
Hi Patrick,
Please see inline.
Regards,
Mridul
On Wed, Jul 2, 2014 at 10:52 AM, Patrick Wendell wrote:
>> b) Instead of pulling this information, push it to executors as part
>> of task submission. (What Patrick mentioned ?)
>> (1) a.1 from above is still an issue for this.
>
> I don't under
Do note that your solution of using broadcast to send the map tasks is very
>> similar to how the executor returns the result of a task when it's too big
>> for akka. We were thinking of refactoring this too, as using the block
>> manager has much higher latency than a dir
different workers
requesting for the output statuses for shuffle (after map) - so not
sure if back pressure buffers, etc would help.
Regards,
Mridul
On Mon, Jun 30, 2014 at 11:07 PM, Mridul Muralidharan wrote:
> Hi,
>
> While sending map output tracker result, the same serialized byte
&
Hi,
While sending map output tracker result, the same serialized byte
array is sent multiple times - but the akka implementation copies it
to a private byte array within ByteString for each send.
Caching a ByteString instead of Array[Byte] did not help, since akka
does not support special casing
[
https://issues.apache.org/jira/browse/SPARK-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045433#comment-14045433
]
Mridul Muralidharan commented on SPARK-2294:
I agree; We should bum
[
https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043088#comment-14043088
]
Mridul Muralidharan commented on SPARK-2268:
That is not because of this
[
https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043071#comment-14043071
]
Mridul Muralidharan commented on SPARK-2268:
Setting priority for shut
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-1476:
---
Attachment: 2g_fix_proposal.pdf
Proposal detailing the work we have done on this
dul,
>
> Can you comment a little bit more on this issue? We are running into the
> same stack trace but not sure whether it is just different Spark versions
> on each cluster (doesn't seem likely) or a bug in Spark.
>
> Thanks.
>
>
>
> On Sat, May 17, 2014 at 4:41 AM
[
https://issues.apache.org/jira/browse/SPARK-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040094#comment-14040094
]
Mridul Muralidharan commented on SPARK-2223:
I usually do :
$ mvn -P
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040085#comment-14040085
]
Mridul Muralidharan commented on SPARK-2089:
A few of things to be kep
[
https://issues.apache.org/jira/browse/SPARK-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039742#comment-14039742
]
Mridul Muralidharan edited comment on SPARK-704 at 6/21/14 9:1
801 - 900 of 1557 matches
Mail list logo
