Wow, it all works now! Thanks, Imran!
In case someone else finds this useful, here are the additional classes
that I had to register (in addition to my application specific classes):
val tuple3ArrayClass = classOf[Array[Tuple3[Any, Any, Any]]]
val anonClass = Class.forName("scala.reflect.ClassTag$$anon$1")
val javaClassClass = classOf[java.lang.Class[Any]]
arun
On Tue, Apr 14, 2015 at 6:23 PM, Imran Rashid <iras...@cloudera.com> wrote:
> hmm, I dunno why IntelliJ is unhappy, but you can always fall back to
> getting a class from the String:
>
> Class.forName("scala.reflect.ClassTag$$anon$1")
>
> perhaps the class is package private or something, and the repl somehow
> subverts it ...
>
> On Tue, Apr 14, 2015 at 5:44 PM, Arun Lists <lists.a...@gmail.com> wrote:
>
>> Hi Imran,
>>
>> Thanks for the response! However, I am still not there yet.
>>
>> In the Scala interpreter, I can do:
>>
>> scala> classOf[scala.reflect.ClassTag$$anon$1]
>>
>> but when I try to do this in my program in IntelliJ, it indicates an
>> error:
>>
>> Cannot resolve symbol ClassTag$$anon$1
>>
>> Hence I am not any closer to making this work. If you have any further
>> suggestions, they would be most welcome.
>>
>> arun
>>
>>
>> On Tue, Apr 14, 2015 at 2:33 PM, Imran Rashid <iras...@cloudera.com>
>> wrote:
>>
>>> Hi Arun,
>>>
>>> It can be hard to use kryo with required registration because of issues
>>> like this -- there isn't a good way to register all the classes that you
>>> need transitively. In this case, it looks like one of your classes has a
>>> reference to a ClassTag, which in turn has a reference to some anonymous
>>> inner class. I'd suggest
>>>
>>> (a) figuring out whether you really want to be serializing this thing --
>>> its possible you're serializing an RDD which keeps a ClassTag, but normally
>>> you wouldn't want to serialize your RDDs
>>> (b) you might want to bring this up w/ chill -- spark offloads most of
>>> the kryo setup for all the scala internals to chill, I'm surprised they
>>> don't handle this already. Looks like they still handle ClassManifests
>>> which are from pre-scala 2.10:
>>> https://github.com/twitter/chill/blob/master/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L189
>>>
>>> (c) you can always register these classes yourself, despite the crazy
>>> names, though you'll just need to knock these out one-by-one:
>>>
>>> scala> classOf[scala.reflect.ClassTag$$anon$1]
>>>
>>> res0: Class[scala.reflect.ClassTag[T]{def unapply(x$1:
>>> scala.runtime.BoxedUnit): Option[_]; def arrayClass(x$1: Class[_]):
>>> Class[_]}] = class scala.reflect.ClassTag$$anon$1
>>>
>>> On Mon, Apr 13, 2015 at 6:09 PM, Arun Lists <lists.a...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to register classes with KryoSerializer. This has worked
>>>> with other programs. Usually the error messages are helpful in indicating
>>>> which classes need to be registered. But with my current program, I get the
>>>> following cryptic error message:
>>>>
>>>> *Caused by: java.lang.IllegalArgumentException: Class is not
>>>> registered: scala.reflect.ClassTag$$anon$1*
>>>>
>>>> *Note: To register this class use:
>>>> kryo.register(scala.reflect.ClassTag$$anon$1.class);*
>>>>
>>>> How do I find out which class needs to be registered? I looked at my
>>>> program and registered all classes used in RDDs. But clearly more classes
>>>> remain to be registered if I can figure out which classes.
>>>>
>>>> Thanks for your help!
>>>>
>>>> arun
>>>>
>>>>
>>>>
>>>
>>
>
