Re: How to access global kryo instance?

I see -- the answer is no, we do currently not use an object pool, but instead just try to create it less frequently (typically one SerializerInstance per partition). For instance, you could do rdd.mapPartitions { partitionIterator => val kryo = SparkEnv.get.serializer.newKryo() partitionItera

Re: How to access global kryo instance?

On Tue, Jan 7, 2014 at 2:52 AM, Aaron Davidson wrote: > Please take a look at the source code -- it's relatively friendly, and > very useful for digging into Spark internals! > (KryoSerializer

Re: How to access global kryo instance?

Please take a look at the source code -- it's relatively friendly, and very useful for digging into Spark internals! (KryoSerializer ) As you can see, a Kryo instance is avai

Re: How to access global kryo instance?

In a map closure, I could use: val ser = SparkEnv.get.serializer.asInstanceOf[KryoSerializer] But how to get the instance of Kryo that spark uses from ser? On Tue, Jan 7, 2014 at 1:04 AM, Aaron Davidson wrote: > I believe SparkEnv.get.serializer would return the serializer created from > the

Re: How to access global kryo instance?

I believe SparkEnv.get.serializer would return the serializer created from the "spark.serializer" property. You can also obtain a Kryo serializer directly via it's no-arg constructor (it still invokes your spark.kryo.registrator): val serializer = new KryoSerializer() but this could have some over

How to access global kryo instance?

Hi, Is there a way to access the global kryo instance created by spark? I'm referring to the one which is passed to registerClasses() in a KryoRegistrator sub class. I'd like to access this kryo instance inside a map closure, so it should be accessible from thw workers side too.