On Sat, Mar 22, 2014 at 7:45 PM, andy petrella wrote:
> Dear,
> I'm pretty much following the Pascal's advices, since I've myseelf
> encoutered some problems with implicits (when playing the same kind of game
> with my Neo4J Scala API).
>
> Nevertheless, one remark regarding the serialization, the
Ok guys, I've read spark code a bit deeper on serialization side...
You're right, Java & Kryo serialization are runtime only so yes this isn't
really a problem.
A few weeks ago, I've studied a bit how we could integrate Pickling to
spark but currently it's not really possible as Pickling is based
Hi Pascal,
Thanks for the input. I think we are going to be okay here since, as Koert
said, the current serializers use runtime type information. We could also
keep at ClassTag around for the original type when the RDD was created.
Good things to be aware of though.
Michael
On Sat, Mar 22, 20
On Sat, Mar 22, 2014 at 8:38 PM, David Hall wrote:
> On Sat, Mar 22, 2014 at 8:59 AM, Pascal Voitot Dev <
> pascal.voitot@gmail.com> wrote:
>
> > The problem I was talking about is when you try to use typeclass
> converters
> > and make them contravariant/covariant for input/output. Something
On Sat, Mar 22, 2014 at 8:59 AM, Pascal Voitot Dev <
pascal.voitot@gmail.com> wrote:
> The problem I was talking about is when you try to use typeclass converters
> and make them contravariant/covariant for input/output. Something like:
>
> Reader[-I, +O] { def read(i:I): O }
>
> Doing this, y
Dear,
I'm pretty much following the Pascal's advices, since I've myseelf
encoutered some problems with implicits (when playing the same kind of game
with my Neo4J Scala API).
Nevertheless, one remark regarding the serialization, the lost of data
shouldn't arrive in the case whenimplicit typeclasse
i believe kryo serialization uses runtime class, not declared class
we have no issues serializing covariant scala lists
On Sat, Mar 22, 2014 at 11:59 AM, Pascal Voitot Dev <
pascal.voitot@gmail.com> wrote:
> On Sat, Mar 22, 2014 at 3:45 PM, Michael Armbrust >wrote:
>
> > >
> > > From my exp
On Sat, Mar 22, 2014 at 3:45 PM, Michael Armbrust wrote:
> >
> > From my experience, covariance often becomes a pain when dealing with
> > serialization/deserialization (I've experienced a few cases while
> > developing play-json & datomisca).
> > Moreover, if you have implicits, variance often be
>
> From my experience, covariance often becomes a pain when dealing with
> serialization/deserialization (I've experienced a few cases while
> developing play-json & datomisca).
> Moreover, if you have implicits, variance often becomes a headache...
This is exactly the kind of feedback I was hop
Hi,
Covariance always seems like a good idea at first but you must be really
careful as it always has unexpected consequences...
>From my experience, covariance often becomes a pain when dealing with
serialization/deserialization (I've experienced a few cases while
developing play-json & datomisca
That would be awesome. I support this!
On Fri, Mar 21, 2014 at 7:28 PM, Michael Armbrust
wrote:
> Hey Everyone,
>
> Here is a pretty major (but source compatible) change we are considering
> making to the RDD API for 1.0. Java and Python APIs would remain the same,
> but users of Scala would lik
Hey Everyone,
Here is a pretty major (but source compatible) change we are considering
making to the RDD API for 1.0. Java and Python APIs would remain the same,
but users of Scala would likely need to use less casts. This would be
especially true for libraries whose functions take RDDs as param
12 matches
Mail list logo
