On Thu, Nov 1, 2012 at 11:26 AM, Christoph Engelbert <[email protected]> wrote: > Am 01.11.2012 18:35, schrieb Tatu Saloranta: >> On Thu, Nov 1, 2012 at 10:06 AM, Christoph Engelbert >> <[email protected]> wrote: >>> Am 01.11.2012 17:48, schrieb Roman Levenstein: >>>> Hi, >>>> >>>> On Thu, Nov 1, 2012 at 4:45 PM, Tatu Saloranta <[email protected]> >>>> wrote: >>>>> On Thu, Nov 1, 2012 at 8:37 AM, Christoph Engelbert >>>>> <[email protected]> wrote: >>>>>> Am 01.11.2012 16:35, schrieb Tatu Saloranta: >>>>>>> On Thu, Nov 1, 2012 at 2:53 AM, Christoph Engelbert >>>>>>> <[email protected]> wrote: >>>>>>>> Am 30.10.2012 18:01, schrieb Tatu Saloranta: >>>>>>>>> You may want to see how serializer/deserializer in >>>>>>>>> >>>>>>>>> https://github.com/eishay/jvm-serializers >>>>>>>>> >>>>>>>>> is done -- Martin wrote it (or at least modified & has maintained it), >>>>>>>>> so it should be along best practices. >>>>>>>>> >>>>>>>>> -+ Tatu +- >>>>>>>> This is only best practice for non concurrent access. The unittests >>>>>>>> are executed one by one so the implementation using one byte array >>>>>>>> is not a problem but this cannot be threadsafe. >>>>>>>> >>>>>>>> This applies to this one as well. Never use this implementation in >>>>>>>> concurrent environments :-) >>>>>>>> https://github.com/eishay/jvm-serializers/blob/kannan/tpc/src/serializers/Kryo.java >>>>>>> Correct. What I meant was it was right way to do it for the use case >>>>>>> in question. Which granted requires one to understand the use case... >>>>>>> I forgot that this particular test does not do multi-threaded >>>>>>> processing like others do. >>>>>>> >>>>>>> It actually is one of the things for jvm-serializers to consider; >>>>>>> efficient reuse techniques vary a lot between codecs. >>>>>>> >>>>>>> -+ Tatu +- >>>>>> I guess there's still the problem about the kryo instance itself. On >>>>>> googlecode it is marked that the kryo instance itself is not >>>>>> threadsafe. Anyone has some more informations about that? Tatu? :-) >>>>> No, I have only tried to use it without success once or twice, outside >>>>> of jvm-serializers. >>>>> My weapon of choice if Smile+Jakcson. :-D >>>>> >>>>> But Kryo author (Nate something?) should know -- I can try to dig >>>>> contact info, point him to this mailing list. >>>>> >>>>> -+ Tatu +- >>>> I'm one of Kryo's contributors. >>>> Kryo instances do keep some context information during serialization >>>> and deserialization. Therefore, the same instance cannot be safely >>>> used at the same time with multiple input or output streams for >>>> performing simultaneous (de) serializations. And using the same >>>> instance with multiple threads can result in even more problems. >>>> >>>> Therefore, when I need to use Kryo in multi-threaded environment, I >>>> usually make sure that each thread gets its own instance of Kryo at >>>> least for the duration of the operation. The reduce the cost of Kryo >>>> instances creation I usually use a pool of such instances. Every time >>>> a thread needs to perform a (de) serialization operation, it would get >>>> an instance from the pool, perform the operation and return the >>>> instance back to the pool. Alternatively, one could use ThreadLocal >>>> Kryo instances if applicable. >>>> >>>> -Roman >>> Ok that is a nice information. So I would suggest to use >>> concurrencylevel to instantiate a bunch of kryo instances and queue >>> them. If non of the preinstantiated kryo instances is available than >>> the (de-)serialization must wait until one's free again (but that >>> would mean the concurrencylevel is choosen to low ;-)). >> Instead of queue which needs synchronization, it may be better to use >> ThreadLocal with soft reference. This requires no synchronization for >> access, and also cleans up instances if needed for GC. This can make a >> big difference for multi-threaded access due to reduced lock >> contestion. >> And the details can be hidden behind a shared factory if that simplifies >> things. >> >> -+ Tatu +- > > But I guess I would need to register classes in all of the kryo > instances at the same time, or how creates kryo ids for the > different classes?
They'd need to get registered when instances are created (or via factory used to create them). Same way I think as what queue approach would do, except can't do eagerly (which may or may not be done with queues). This is why a separate factory would make sense, and ThreadLocal+SoftReference is just the mechanism for retaining them for possible reuse. -+ Tatu +-
