[
https://issues.apache.org/jira/browse/SPARK-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039742#comment-14039742
]
Mridul Muralidharan commented on SPARK-704:
---
If remote node goes
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039236#comment-14039236
]
Mridul Muralidharan commented on SPARK-2089:
[~pwendell] SplitInfo is
[
https://issues.apache.org/jira/browse/SPARK-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039217#comment-14039217
]
Mridul Muralidharan commented on SPARK-2223:
[~tgraves] You could try run
On Wed, Jun 18, 2014 at 6:19 PM, Surendranauth Hiraman
wrote:
> Patrick,
>
> My team is using shuffle consolidation but not speculation. We are also
> using persist(DISK_ONLY) for caching.
Use of shuffle consolidation is probably what is causing the issue.
Would be good idea to try again with th
[
https://issues.apache.org/jira/browse/SPARK-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033625#comment-14033625
]
Mridul Muralidharan commented on SPARK-1353:
This is due to limitatio
In that case, does it work if you use snappy instead of lzf ?
Regards,
Mridul
On Mon, Jun 16, 2014 at 7:34 AM, gchen wrote:
> To anyone who is interested in this issue, the root cause if from a third
> party code com.ning.compress.lzf.impl.UnsafeChunkEncoderBE class since they
> have a broken
[
https://issues.apache.org/jira/browse/SPARK-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027808#comment-14027808
]
Mridul Muralidharan commented on SPARK-2018:
Ah ! This is an interesting
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026397#comment-14026397
]
Mridul Muralidharan commented on SPARK-2089:
preferredNodeLocationData
[
https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14021020#comment-14021020
]
Mridul Muralidharan commented on SPARK-2064:
Ah, I assumed there w
[
https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14021011#comment-14021011
]
Mridul Muralidharan commented on SPARK-2064:
I am probably missing the in
[
https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14021008#comment-14021008
]
Mridul Muralidharan commented on SPARK-2064:
Unfortunately OOM is a very
[
https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020936#comment-14020936
]
Mridul Muralidharan commented on SPARK-2064:
It is 100 MB (or more) of me
[
https://issues.apache.org/jira/browse/SPARK-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020789#comment-14020789
]
Mridul Muralidharan commented on SPARK-2064:
Depending on how long a job
[
https://issues.apache.org/jira/browse/SPARK-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019427#comment-14019427
]
Mridul Muralidharan commented on SPARK-2045:
The plan Tom and I had wa
[
https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019394#comment-14019394
]
Mridul Muralidharan commented on SPARK-2017:
Currently, for our jobs, I
[
https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012296#comment-14012296
]
Mridul Muralidharan edited comment on SPARK-1956 at 5/29/14 11:4
[
https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012296#comment-14012296
]
Mridul Muralidharan commented on SPARK-1956:
We have not taken a call on
[
https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011741#comment-14011741
]
Mridul Muralidharan commented on SPARK-1956:
shuffle consolidation MUST
[
https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001381#comment-14001381
]
Mridul Muralidharan commented on SPARK-1855:
Currently, StorageL
[
https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001377#comment-14001377
]
Mridul Muralidharan commented on SPARK-1855:
Did not realize that
[
https://issues.apache.org/jira/browse/SPARK-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001378#comment-14001378
]
Mridul Muralidharan commented on SPARK-1855:
matei.zaha...@gmail.c
[
https://issues.apache.org/jira/browse/SPARK-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001376#comment-14001376
]
Mridul Muralidharan commented on SPARK-1767:
Did not realize that
DISK_2 or construct your own StorageLevel with your own custom
> replication factor.
>
> BTW you guys should probably have this discussion on the JIRA rather than the
> dev list; I think the replies somehow ended up on the dev list.
>
> Matei
>
> On May 17, 2014, at 1:36
intaining backward
> compatibility will be high.
>
> We just need to make an informed decision to live with that cost, not hand
> wave it away.
>
> Regards
> Mridul
>
>> there is anything apparent now that is expected to require such disruptive
>> changes if we wer
ust need to make an informed decision to live with that cost, not hand
wave it away.
Regards
Mridul
> there is anything apparent now that is expected to require such disruptive
> changes if we were to commit to the current release candidate as our
> guaranteed 1.0.0 baseline.
>
>
May 17, 2014, at 10:32 AM, Kan Zhang wrote:
>
> > +1 on the running commentary here, non-binding of course :-)
> >
> >
> > On Sat, May 17, 2014 at 8:44 AM, Andrew Ash
> wrote:
> >
> >> +1 on the next release feeling more like a 0.10 than a 1.0
> >
cle
Regards
Mridul
> If you can tell me about specific changes in the current release
candidate
> that occasion new arguments for why a 1.0 release is an unacceptable idea,
> then I'm listening.
>
>
> On Sat, May 17, 2014 at 11:59 AM, Mridul Muralidharan wrote:
>
> >
muzzle the discussion.
Regards
Mridul
> issue, and what I am asking, is which pending bug fixes does anyone
> anticipate will require breaking the public API guaranteed in rc9
>
>
> On Sat, May 17, 2014 at 9:44 AM, Mridul Muralidharan wrote:
>
> > We made incompatible api chan
ng changes, now would be a
> good time to set me straight.
>
>
> On Sat, May 17, 2014 at 4:26 AM, Mridul Muralidharan >wrote:
>
> > I had echoed similar sentiments a while back when there was a discussion
> > around 0.10 vs 1.0 ... I would have preferred 0.10 to stabil
I suspect this is an issue we have fixed internally here as part of a
larger change - the issue we fixed was not a config issue but bugs in spark.
Unfortunately we plan to contribute this as part of 1.1
Regards,
Mridul
On 17-May-2014 4:09 pm, "sam (JIRA)" wrote:
> sam created SPARK-1867:
>
I had echoed similar sentiments a while back when there was a discussion
around 0.10 vs 1.0 ... I would have preferred 0.10 to stabilize the api
changes, add missing functionality, go through a hardening release before
1.0
But the community preferred a 1.0 :-)
Regards,
Mridul
On 17-May-2014 3:19
we have
> enough memory to use. We need to investigate more to find a good
> solution. -Xiangrui
>
> On Fri, May 16, 2014 at 4:00 PM, Mridul Muralidharan
> wrote:
> > Effectively this is persist without fault tolerance.
> > Failure of any node means complete lack o
[
https://issues.apache.org/jira/browse/SPARK-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000716#comment-14000716
]
Mridul Muralidharan commented on SPARK-1865:
You could also modify
Can you try moving your mapPartitions to another class/object which is
referenced only after sc.addJar ?
I would suspect CNFEx is coming while loading the class containing
mapPartitions before addJars is executed.
In general though, dynamic loading of classes means you use reflection to
instantia
[
https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000580#comment-14000580
]
Mridul Muralidharan commented on SPARK-1849:
You are missing my p
So was rc5 cancelled ? Did not see a note indicating that or why ... [1]
- Mridul
[1] could have easily missed it in the email storm though !
On Thu, May 15, 2014 at 1:32 AM, Patrick Wendell wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.0.0!
>
> This pa
[
https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000397#comment-14000397
]
Mridul Muralidharan commented on SPARK-1849:
Looks like textFile is prob
Effectively this is persist without fault tolerance.
Failure of any node means complete lack of fault tolerance.
I would be very skeptical of truncating lineage if it is not reliable.
On 17-May-2014 3:49 am, "Xiangrui Meng (JIRA)" wrote:
> Xiangrui Meng created SPARK-1855:
>
Hi Sandy,
I assume you are referring to caching added to datanodes via new caching
api via NN ? (To preemptively mmap blocks).
I have not looked in detail, but does NN tell us about this in block
locations?
If yes, we can simply make those process local instead of node local for
executors on th
[
https://issues.apache.org/jira/browse/SPARK-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996390#comment-13996390
]
Mridul Muralidharan commented on SPARK-1813:
Writing a KryoRegistrator is
[
https://issues.apache.org/jira/browse/SPARK-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995600#comment-13995600
]
Mridul Muralidharan commented on SPARK-1772:
BTW, for specific case of
On a slightly related note (apologies Soren for hijacking the thread),
Reynold how much better is kryo from spark's usage point of view
compared to the default java serialization (in general, not for
closures) ?
The numbers on kyro site are interesting, but since you have played
the most with kryo
[
https://issues.apache.org/jira/browse/SPARK-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988868#comment-13988868
]
Mridul Muralidharan commented on SPARK-1706:
Oh my, this was supposed t
[
https://issues.apache.org/jira/browse/SPARK-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988807#comment-13988807
]
Mridul Muralidharan commented on SPARK-1697:
I would suspect this is du
[
https://issues.apache.org/jira/browse/SPARK-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-1606:
---
Comment: was deleted
(was: Crap, got to this too late.
We really should not have
[
https://issues.apache.org/jira/browse/SPARK-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988756#comment-13988756
]
Mridul Muralidharan commented on SPARK-1606:
Crap, got to this too late
[
https://issues.apache.org/jira/browse/BOOKKEEPER-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated BOOKKEEPER-648:
---
Assignee: (was: Mridul Muralidharan)
> BasicJMSTest fai
[
https://issues.apache.org/jira/browse/BOOKKEEPER-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated BOOKKEEPER-560:
---
Assignee: (was: Mridul Muralidharan)
> Create readme for hed
[
https://issues.apache.org/jira/browse/SPARK-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-1587.
Resolution: Fixed
Fixed, https://github.com/apache/spark/pull/504
> Fix thr
[
https://issues.apache.org/jira/browse/SPARK-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981321#comment-13981321
]
Mridul Muralidharan commented on SPARK-1586:
Immediate issues fixed th
[
https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981313#comment-13981313
]
Mridul Muralidharan commented on SPARK-1576:
There is a misunderstanding
[
https://issues.apache.org/jira/browse/SPARK-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978827#comment-13978827
]
Mridul Muralidharan commented on SPARK-1588:
Apparently, SPARK_YARN_USER
Sorry, I misread - I meant SPARK_JAVA_OPTS - not JAVA_OPTS.
See here : https://issues.apache.org/jira/browse/SPARK-1588
Regards,
Mridul
On Wed, Apr 23, 2014 at 6:37 PM, Mridul Muralidharan wrote:
> This breaks all existing jobs which are not using spark-submit.
> The consensus was not to
This breaks all existing jobs which are not using spark-submit.
The consensus was not to break compatibility unless there was an overriding
reason to do so
On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" wrote:
>
> [
> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.p
[
https://issues.apache.org/jira/browse/SPARK-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978095#comment-13978095
]
Mridul Muralidharan commented on SPARK-1588:
Noticed this specificall
Mridul Muralidharan created SPARK-1588:
--
Summary: SPARK_JAVA_OPTS is not getting propagated
Key: SPARK-1588
URL: https://issues.apache.org/jira/browse/SPARK-1588
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-1586:
--
Assignee: Mridul Muralidharan
> Fix issues with spark development un
Mridul Muralidharan created SPARK-1587:
--
Summary: Fix thread leak in spark
Key: SPARK-1587
URL: https://issues.apache.org/jira/browse/SPARK-1587
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-1587:
--
Assignee: Mridul Muralidharan
> Fix thread leak in sp
Mridul Muralidharan created SPARK-1586:
--
Summary: Fix issues with spark development under windows
Key: SPARK-1586
URL: https://issues.apache.org/jira/browse/SPARK-1586
Project: Spark
nto memory at once. The ShuffledRDD is agnostic to what goes inside P.
>
> On Sun, Apr 20, 2014 at 11:36 AM, Mridul Muralidharan wrote:
>
>> An iterator does not imply data has to be memory resident.
>> Think merge sort output as an iterator (disk backed).
>>
>> Tom is
An iterator does not imply data has to be memory resident.
Think merge sort output as an iterator (disk backed).
Tom is actually planning to work on something similar with me on this
hopefully this or next month.
Regards,
Mridul
On Sun, Apr 20, 2014 at 11:46 PM, Sandy Ryza wrote:
> Hey all,
>
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972978#comment-13972978
]
Mridul Muralidharan commented on SPARK-1476:
[~matei] We are having
[
https://issues.apache.org/jira/browse/SPARK-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972864#comment-13972864
]
Mridul Muralidharan commented on SPARK-1524:
The expectation is to fall
[
https://issues.apache.org/jira/browse/SPARK-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968390#comment-13968390
]
Mridul Muralidharan commented on SPARK-1453:
(d) becomes relevant in cas
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968147#comment-13968147
]
Mridul Muralidharan commented on SPARK-1476:
[~pwendell] IMO both
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-1476:
--
Assignee: Mridul Muralidharan
> 2GB limit in spark for blo
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967854#comment-13967854
]
Mridul Muralidharan edited comment on SPARK-1476 at 4/13/14 2:4
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967854#comment-13967854
]
Mridul Muralidharan commented on SPARK-1476:
There are multiple issue
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-1476:
---
Fix Version/s: 1.1.0
> 2GB limit in spark for blo
[
https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967419#comment-13967419
]
Mridul Muralidharan commented on SPARK-1476:
WIP Proposal:
- All refere
Mridul Muralidharan created SPARK-1476:
--
Summary: 2GB limit in spark for blocks
Key: SPARK-1476
URL: https://issues.apache.org/jira/browse/SPARK-1476
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967193#comment-13967193
]
Mridul Muralidharan commented on SPARK-1453:
The timeout gets hit only
[
https://issues.apache.org/jira/browse/SPARK-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967185#comment-13967185
]
Mridul Muralidharan commented on SPARK-542:
---
Spark uses only hostnames -
[
https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966332#comment-13966332
]
Mridul Muralidharan commented on SPARK-1391:
Another place where thi
, which is stored in a remote cluster or machines. And the
> goal is to load the remote raw data only once?
>
> Haoyuan
>
>
> On Sat, Apr 5, 2014 at 4:30 PM, Mridul Muralidharan >wrote:
>
> > Hi,
> >
> > We have a requirement to use a (potential) ephemeral
[
https://issues.apache.org/jira/browse/SPARK-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-1393.
Resolution: Fixed
> fix computePreferredLocations signature to not depend
[
https://issues.apache.org/jira/browse/SPARK-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961278#comment-13961278
]
Mridul Muralidharan commented on SPARK-1393:
Merged https://github.com/ap
Hi,
We have a requirement to use a (potential) ephemeral storage, which
is not within the VM, which is strongly tied to a worker node. So
source of truth for a block would still be within spark; but to
actually do computation, we would need to copy data to external device
(where it might lie aro
[
https://issues.apache.org/jira/browse/SPARK-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-1393:
---
Description: computePreferredLocations in
core/src/main/scala/org/apache/spark
Mridul Muralidharan created SPARK-1393:
--
Summary: fix computePreferredLocations signature to not depend on
underlying implementation
Key: SPARK-1393
URL: https://issues.apache.org/jira/browse/SPARK-1393
[
https://issues.apache.org/jira/browse/SPARK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955997#comment-13955997
]
Mridul Muralidharan commented on SPARK-1350:
You mistook my question; I m
[
https://issues.apache.org/jira/browse/SPARK-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955978#comment-13955978
]
Mridul Muralidharan commented on SPARK-1350:
We will need a way to confi
>> You can unsubscribe yourself from any of these sources, right?
>>
>> - Patrick
>>
>>
>> On Sat, Mar 29, 2014 at 11:05 AM, Mridul Muralidharan
>> wrote:
>>
>>> Hi,
>>>
>>> So we are now receiving updates from three source
Hi,
So we are now receiving updates from three sources for each change to the PR.
While each of them handles a corner case which others might miss,
would be great if we could minimize the volume of duplicated
communication.
Regards,
Mridul
Got some 100 odd mails from jenkins (?) with "Can one of the admins
verify this patch?"
Part of upgrade or some other issue ?
Significantly reduced the snr of my inbox !
Regards,
Mridul
ly long running job (30 mins+) working on non
trivial dataset will fail due to accumulated failures in spark.
Regards,
Mridul
>
> TD
>
>
>
>
> On Tue, Mar 25, 2014 at 8:44 PM, Mridul Muralidharan wrote:
>
>> Forgot to mention this in the earlier request for PR'
ut not pulled into branch 0.9. I am
>> not sure it is a good idea to pull that in. We can pull those changes later
>> for 0.9.2 if required.
>>
>> TD
>>
>>
>>
>>
>> On Tue, Mar 25, 2014 at 8:44 PM, Mridul Muralidharan > >wrote:
>>
>
Forgot to mention this in the earlier request for PR's.
If there is another RC being cut, please add
https://github.com/apache/spark/pull/159 to it too (if not done
already !).
Thanks,
Mridul
On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das
wrote:
> Hello everyone,
>
> Since the release of Spark
ected to be release around end of April (not too far
> ;) ).
>
> TD
>
>
> On Wed, Mar 19, 2014 at 5:57 PM, Mridul Muralidharan wrote:
>
>> Would be great if the garbage collection PR is also committed - if not
>> the whole thing, atleast the part to unpersist broadca
Would be great if the garbage collection PR is also committed - if not
the whole thing, atleast the part to unpersist broadcast variables
explicitly would be great.
Currently we are running with a custom impl which does something
similar, and I would like to move to standard distribution for that.
;t work anymore if
> we standardize on SBT. These have no obvious work around at this point
> as far as I see.
>
> - Patrick
>
> On Wed, Feb 26, 2014 at 7:09 PM, Mridul Muralidharan wrote:
>> On Feb 26, 2014 11:12 PM, "Patrick Wendell" wrote:
>>>
>&g
tions to using sbt or maven !
> >> Too many exclude versions, pinned versions, etc would just make things
> >> unmanageable in future.
> >>
> >>
> >> Regards,
> >> Mridul
> >>
> >>
> >>
> >>
> >> On W
others. (#NelsonMandela)
>
>> On Feb 25, 2014, at 6:50 PM, Mridul Muralidharan wrote:
>>
>>> On Wed, Feb 26, 2014 at 5:31 AM, Patrick Wendell wrote:
>>> Evan - this is a good thing to bring up. Wrt the shader plug-in -
>>> right now we don't actuall
On Wed, Feb 26, 2014 at 5:31 AM, Patrick Wendell wrote:
> Evan - this is a good thing to bring up. Wrt the shader plug-in -
> right now we don't actually use it for bytecode shading - we simply
> use it for creating the uber jar with excludes (which sbt supports
> just fine via assembly).
Not re
Hi,
I have not looked into why this would be needed, but given it is
needed, I added a couple of comments to the PR.
Overall, it looks promising.
Regards,
Mridul
On Tue, Feb 25, 2014 at 8:05 AM, 尹绪森 wrote:
> Hi community,
>
> As I moving forward to write a LDA (Latent Dirichlet Allocation) t
Curious, what was the issue ?
- Mridul
On Sun, Feb 23, 2014 at 11:41 PM, Nan Zhu wrote:
> OK, I know where I was wrong
>
>
> Best,
>
> --
> Nan Zhu
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Sunday, February 23, 2014 at 12:50 PM, Nan Zhu wrote:
>
>> String, it should be ge
e (sparkbank)? Curious to know how you'd decide what
> should go where.
>
> Amandeep
>
> > On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan
> wrote:
> >
> > Hi,
> >
> > Over the past few months, I have seen a bunch of pull requests which
> have
Hi,
Over the past few months, I have seen a bunch of pull requests which have
extended spark api ... most commonly RDD itself.
Most of them are either relatively niche case of specialization (which
might not be useful for most cases) or idioms which can be expressed
(sometimes with minor perf p
I am not sure if this is resolved now - but maven was better at
building the assembly jars compared to sbt.
To the point where I stopped using sbt due to unpredictable order in
which it unjars the dependencies to create the assembled jar (we do
have quite a lot of conflicting classes in our depende
901 - 1000 of 1557 matches
Mail list logo
