[
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474333#comment-16474333
]
ASF GitHub Bot commented on METRON-1551:
----------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1012
Ok, this looks great, one request though. It appears that, at least for
the profiler, it is required that objects used in Stellar be Java serializable
if they aren't listed in that exception list.
We ensure that the objects are kryo serializable
[here](https://github.com/apache/metron/blob/91a017b6dcefc250bfd67cecf9803cb59015d213/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/StellarProcessorUtils.java#L64)
Can we also make a blurb that they're java serializable? I know that
OnlineStatisticsProvider will not be java serializable due to its dependence on
t-digest, so maybe make that check optional but on by default?
> Profiler Should Not Use Java Serialization
> ------------------------------------------
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the
> following error happens. This can occur when the number of workers is
> greater than 1.
> The topology should not be using Java serialization for serializing tuple
> values as this will negatively impact performance.
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> ... 6 more
> Caused by: java.io.NotSerializableException:
> org.apache.metron.common.configuration.profiler.ProfileResult
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> ~[?:1.8.0_162]
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> ~[?:1.8.0_162]
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> ~[?:1.8.0_162]
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> ~[?:1.8.0_162]
> at
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> ... 6 more
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
