Hi Aaron, While we are debugging the test failures on dev, I didn't fully understand the last question in your email. Looks like you mentioned that the workaround for adding the @FieldSerializer to the field is working. Are you looking for an alternative to this workaround and trying to set a default serializer for all fields of type Instant (the field type you mentioned in the earlier email) so that you don't have to set the annotation each time?
Thanks On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com> wrote: > I sent an e-mail to the dev list...but have not heard any responses. How > active is that list? Do you know? Basically, I have a workaround that > will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2 > or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I > have two options left: go it alone and try to update Kryo in the apex-core, > or simply change the default serializer. The hiccup I am running into is > that I don't see how (from the documentation) to set the default > serializer...perhaps I am missing it. Can someone point me to where that > is in the docs? > > On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> Ah, you make a good point about the dev list...I am using IntelliJ on a >> Mac. When I tried building without any changes, I also encountered >> failures. I can re-run the non-modified sources to do a diff and see if >> the errors were different... >> >> Sent from my iPhone >> >> On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.imman...@gmail.com> >> wrote: >> >> Aaron, >> >> The question might be best served on the dev@apex.apache.org mailing >> list as you are starting to make changes to the sources. Nothing obvious >> jumps out to be, you don't need hadoop on your system to build the sources >> successfully with all the tests passing. The test failures you are seeing >> may or may not be related to the kryo change. What happens when you try to >> build the original sources with changing the Kryo version? What is your >> environment, mac, linux? >> >> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> I know it has been a little while since I brought this up...I pulled >>> down the 3.7.1-RELEASE branch from GitHub in order to test out the change >>> to Kryo v. 4.0.2 and encountered no problems while building, however, when >>> I try to do mvn install, there are several failed tests. I am hoping, that >>> other than the obvious (to me, anyway) errors such as not finding >>> yarn-site.xml on my laptop which does not have hadoop installed, someone >>> may be able to see what is going on? >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.StramRecoveryTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 >>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest >>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time >>> elapsed: 0.115 sec <<< FAILURE! >>> java.lang.AssertionError: flush count expected:<1> but was:<2> >>> at >>> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) >>> >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.engine.StatsTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 >>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest >>> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) >>> Time elapsed: 3.266 sec <<< FAILURE! >>> java.lang.AssertionError: Validate input port queue size -1 >>> at >>> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) >>> at >>> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.StramMiniClusterTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 >>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest >>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< >>> ERROR! >>> java.io.FileNotFoundException: >>> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml >>> (No such file or directory) >>> at >>> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.StramLocalClusterTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 >>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest >>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: >>> 0.138 sec <<< FAILURE! >>> java.lang.AssertionError: checkpoint path exists >>> at >>> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) >>> >>> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) >>> Time elapsed: 5.191 sec <<< FAILURE! >>> java.lang.AssertionError: >>> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out >>> exists >>> at >>> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.client.AppPackageTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 >>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest >>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time >>> elapsed: 0 sec <<< FAILURE! >>> java.lang.AssertionError: Number of applications expected:<2> but was:<0> >>> at >>> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) >>> >>> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) >>> Time elapsed: 0.001 sec <<< FAILURE! >>> java.lang.AssertionError: Should consist of an app called >>> MyFirstApplication >>> at >>> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) >>> >>> >>> >>> ------------------------------------------------------------------------------- >>> Test set: com.datatorrent.stram.cli.ApexCliTest >>> >>> ------------------------------------------------------------------------------- >>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 >>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest >>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>> Time elapsed: 0.104 sec <<< ERROR! >>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >>> Application Package >>> at >>> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) >>> >>> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) >>> Time elapsed: 0.045 sec <<< FAILURE! >>> org.junit.ComparisonFailure: expected:<[app]-default> but >>> was:<[package]-default> >>> at >>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) >>> >>> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>> Time elapsed: 0.045 sec <<< ERROR! >>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >>> Application Package matching "MyFirstApplication" >>> at >>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) >>> >>> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >>> Time elapsed: 0.058 sec <<< FAILURE! >>> org.junit.ComparisonFailure: expected:<[app]-default> but >>> was:<[package]-default> >>> at >>> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) >>> >>> >>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> >>> wrote: >>> >>>> Thanks, Thomas! I have implemented the fix you suggested >>>> (@FieldSerializer.BInd) and also created a JIRA ticket: >>>> https://issues.apache.org/jira/browse/APEXCORE-816 >>>> >>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: >>>> >>>>> Hi, >>>>> >>>>> It is not necessary to use a different stream codec when you have >>>>> control over the class that is causing the error. You can annotate a >>>>> different field serializer instead, using the @FieldSerializer.Bind >>>>> annotation. Here is an example: >>>>> >>>>> >>>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >>>>> >>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you >>>>> like to create a JIRA for it? ( >>>>> http://apex.apache.org/contributing.html) >>>>> >>>>> Thanks, >>>>> Thomas >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> >>>>> wrote: >>>>> >>>>>> I am writing an application that includes a custom object. One of >>>>>> the fields is of type Instant and seems to not agree with the version of >>>>>> Kryo that is used in Apex (see exception below). I now that kryo 4.X >>>>>> supports serializing Instant fields, but for the life of me, cannot seem >>>>>> to >>>>>> find a way to either get Apex to use a newer version of Kryo. Can anyone >>>>>> help? Perhaps setting a different streamcodec that uses the newer >>>>>> version >>>>>> of kryo would work, but I cannot seem to find an example of how to set a >>>>>> different streamcodec despite references to the two (Kryo and Java) that >>>>>> ship with Apex. >>>>>> >>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>>>>> Catastrophic Error: Execution halted due to Kryo exception! >>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created >>>>>> (missing no-arg constructor): java.time.Instant >>>>>> Serialization trace: >>>>>> systemTime >>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >>>>>> at >>>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>>>>> at >>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >>>>>> at >>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>>>>> at >>>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >>>>>> at >>>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>>> at com.datatorrent.stram.codec.De >>>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >>>>>> at >>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >>>>>> at >>>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >>>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>>>>> at >>>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> M. Aaron Bossert >>>>>> (571) 242-4021 >>>>>> Punch Cyber Analytics Group >>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> M. Aaron Bossert >>>> (571) 242-4021 >>>> Punch Cyber Analytics Group >>>> >>>> >>>> >>> >>> >>> -- >>> >>> M. Aaron Bossert >>> (571) 242-4021 >>> Punch Cyber Analytics Group >>> >>> >>> > > -- > > M. Aaron Bossert > (571) 242-4021 > Punch Cyber Analytics Group > > >
