Re: Kryo registration for Tuples?
Yes you can :) On Wed, Jun 8, 2016 at 6:00 PM, Alexander Pivovarov wrote: > Can I just enable spark.kryo.registrationRequired and look at error > messages to get unregistered classes? > > On Wed, Jun 8, 2016 at 5:52 PM, Reynold Xin wrote: > >> Due to type erasure they have no difference, although watch out for Scala >> tuple serialization. >> >> >> On Wednesday, June 8, 2016, Ted Yu wrote: >> >>> I think the second group (3 classOf's) should be used. >>> >>> Cheers >>> >>> On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov < >>> apivova...@gmail.com> wrote: >>> if my RDD is RDD[(String, (Long, MyClass))] Do I need to register classOf[MyClass] classOf[(Any, Any)] or classOf[MyClass] classOf[(Long, MyClass)] classOf[(String, (Long, MyClass))] ? >>> >
Re: Kryo registration for Tuples?
Can I just enable spark.kryo.registrationRequired and look at error messages to get unregistered classes? On Wed, Jun 8, 2016 at 5:52 PM, Reynold Xin wrote: > Due to type erasure they have no difference, although watch out for Scala > tuple serialization. > > > On Wednesday, June 8, 2016, Ted Yu wrote: > >> I think the second group (3 classOf's) should be used. >> >> Cheers >> >> On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov > > wrote: >> >>> if my RDD is RDD[(String, (Long, MyClass))] >>> >>> Do I need to register >>> >>> classOf[MyClass] >>> classOf[(Any, Any)] >>> >>> or >>> >>> classOf[MyClass] >>> classOf[(Long, MyClass)] >>> classOf[(String, (Long, MyClass))] >>> >>> ? >>> >>> >>
Re: Kryo registration for Tuples?
Due to type erasure they have no difference, although watch out for Scala tuple serialization. On Wednesday, June 8, 2016, Ted Yu wrote: > I think the second group (3 classOf's) should be used. > > Cheers > > On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov > wrote: > >> if my RDD is RDD[(String, (Long, MyClass))] >> >> Do I need to register >> >> classOf[MyClass] >> classOf[(Any, Any)] >> >> or >> >> classOf[MyClass] >> classOf[(Long, MyClass)] >> classOf[(String, (Long, MyClass))] >> >> ? >> >> >
Re: Kryo registration for Tuples?
I think the second group (3 classOf's) should be used. Cheers On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov wrote: > if my RDD is RDD[(String, (Long, MyClass))] > > Do I need to register > > classOf[MyClass] > classOf[(Any, Any)] > > or > > classOf[MyClass] > classOf[(Long, MyClass)] > classOf[(String, (Long, MyClass))] > > ? > >
