Pramod, In a nutshell, yes. I can set a @FieldSerializer to the fields in question...the only one I have run into thus far is Instant...but it would be great to not need to annotate each field this way...Also, I could see this becoming problematic should I encounter more than a couple of fields that are not serializable using the older version of Kryo (but presumably would work with a more current one)...though admitedly, it would not be that big a deal in the interim. It would be much cleaner to have an update to the most current version of Kryo, IMHO...though I understand that is something of a lift. By the way, Kryo 5 just came out yesterday, so if there is to be effort expended, it might be good to go with the latest version...though, heads up, I tried it out and there seem to be quite a few changes that will be needed...new methods, and definitely some replaced ones...
On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <pramod.imman...@gmail.com> wrote: > Hi Aaron, > > While we are debugging the test failures on dev, I didn't fully understand > the last question in your email. Looks like you mentioned that the > workaround for adding the @FieldSerializer to the field is working. Are you > looking for an alternative to this workaround and trying to set a default > serializer for all fields of type Instant (the field type you mentioned in > the earlier email) so that you don't have to set the annotation each time? > > Thanks > > On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> I sent an e-mail to the dev list...but have not heard any responses. How >> active is that list? Do you know? Basically, I have a workaround that >> will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2 >> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I >> have two options left: go it alone and try to update Kryo in the apex-core, >> or simply change the default serializer. The hiccup I am running into is >> that I don't see how (from the documentation) to set the default >> serializer...perhaps I am missing it. Can someone point me to where that >> is in the docs? >> >> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> Ah, you make a good point about the dev list...I am using IntelliJ on a >>> Mac. When I tried building without any changes, I also encountered >>> failures. I can re-run the non-modified sources to do a diff and see if >>> the errors were different... >>> >>> Sent from my iPhone >>> >>> On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.imman...@gmail.com> >>> wrote: >>> >>> Aaron, >>> >>> The question might be best served on the dev@apex.apache.org mailing >>> list as you are starting to make changes to the sources. Nothing obvious >>> jumps out to be, you don't need hadoop on your system to build the sources >>> successfully with all the tests passing. The test failures you are seeing >>> may or may not be related to the kryo change. What happens when you try to >>> build the original sources with changing the Kryo version? What is your >>> environment, mac, linux? >>> >>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> >>> wrote: >>> >>>> I know it has been a little while since I brought this up...I pulled >>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change >>>> to Kryo v. 4.0.2 and encountered no problems while building, however, when >>>> I try to do mvn install, there are several failed tests. I am hoping, that >>>> other than the obvious (to me, anyway) errors such as not finding >>>> yarn-site.xml on my laptop which does not have hadoop installed, someone >>>> may be able to see what is going on? >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.StramRecoveryTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 >>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest >>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time >>>> elapsed: 0.115 sec <<< FAILURE! >>>> java.lang.AssertionError: flush count expected:<1> but was:<2> >>>> at >>>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.engine.StatsTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 >>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest >>>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) >>>> Time elapsed: 3.266 sec <<< FAILURE! >>>> java.lang.AssertionError: Validate input port queue size -1 >>>> at >>>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) >>>> at >>>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.StramMiniClusterTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 >>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest >>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec >>>> <<< ERROR! >>>> java.io.FileNotFoundException: >>>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml >>>> (No such file or directory) >>>> at >>>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.StramLocalClusterTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 >>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest >>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: >>>> 0.138 sec <<< FAILURE! >>>> java.lang.AssertionError: checkpoint path exists >>>> at >>>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) >>>> >>>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) >>>> Time elapsed: 5.191 sec <<< FAILURE! >>>> java.lang.AssertionError: >>>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out >>>> exists >>>> at >>>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.client.AppPackageTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 >>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest >>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time >>>> elapsed: 0 sec <<< FAILURE! >>>> java.lang.AssertionError: Number of applications expected:<2> but >>>> was:<0> >>>> at >>>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) >>>> >>>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) >>>> Time elapsed: 0.001 sec <<< FAILURE! >>>> java.lang.AssertionError: Should consist of an app called >>>> MyFirstApplication >>>> at >>>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------- >>>> Test set: com.datatorrent.stram.cli.ApexCliTest >>>> >>>> ------------------------------------------------------------------------------- >>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 >>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest >>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>>> Time elapsed: 0.104 sec <<< ERROR! >>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >>>> Application Package >>>> at >>>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) >>>> >>>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) >>>> Time elapsed: 0.045 sec <<< FAILURE! >>>> org.junit.ComparisonFailure: expected:<[app]-default> but >>>> was:<[package]-default> >>>> at >>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) >>>> >>>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>>> Time elapsed: 0.045 sec <<< ERROR! >>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >>>> Application Package matching "MyFirstApplication" >>>> at >>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) >>>> >>>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>>> Time elapsed: 0.058 sec <<< FAILURE! >>>> org.junit.ComparisonFailure: expected:<[app]-default> but >>>> was:<[package]-default> >>>> at >>>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) >>>> >>>> >>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> >>>> wrote: >>>> >>>>> Thanks, Thomas! I have implemented the fix you suggested >>>>> (@FieldSerializer.BInd) and also created a JIRA ticket: >>>>> https://issues.apache.org/jira/browse/APEXCORE-816 >>>>> >>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> It is not necessary to use a different stream codec when you have >>>>>> control over the class that is causing the error. You can annotate a >>>>>> different field serializer instead, using the @FieldSerializer.Bind >>>>>> annotation. Here is an example: >>>>>> >>>>>> >>>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >>>>>> >>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would >>>>>> you like to create a JIRA for it? ( >>>>>> http://apex.apache.org/contributing.html) >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com >>>>>> > wrote: >>>>>> >>>>>>> I am writing an application that includes a custom object. One of >>>>>>> the fields is of type Instant and seems to not agree with the version of >>>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X >>>>>>> supports serializing Instant fields, but for the life of me, cannot >>>>>>> seem to >>>>>>> find a way to either get Apex to use a newer version of Kryo. Can >>>>>>> anyone >>>>>>> help? Perhaps setting a different streamcodec that uses the newer >>>>>>> version >>>>>>> of kryo would work, but I cannot seem to find an example of how to set a >>>>>>> different streamcodec despite references to the two (Kryo and Java) that >>>>>>> ship with Apex. >>>>>>> >>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>>>>>> Catastrophic Error: Execution halted due to Kryo exception! >>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created >>>>>>> (missing no-arg constructor): java.time.Instant >>>>>>> Serialization trace: >>>>>>> systemTime >>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >>>>>>> at >>>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>>>>>> at >>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >>>>>>> at >>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>>>>>> at >>>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >>>>>>> at >>>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>>>> at com.datatorrent.stram.codec.De >>>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >>>>>>> at >>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >>>>>>> at >>>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >>>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>>>>>> at >>>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> M. Aaron Bossert >>>>>>> (571) 242-4021 >>>>>>> Punch Cyber Analytics Group >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> M. Aaron Bossert >>>>> (571) 242-4021 >>>>> Punch Cyber Analytics Group >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> M. Aaron Bossert >>>> (571) 242-4021 >>>> Punch Cyber Analytics Group >>>> >>>> >>>> >> >> -- >> >> M. Aaron Bossert >> (571) 242-4021 >> Punch Cyber Analytics Group >> >> >> -- M. Aaron Bossert (571) 242-4021 Punch Cyber Analytics Group
