I sent an e-mail to the dev list...but have not heard any responses. How active is that list? Do you know? Basically, I have a workaround that will deal with those "things" that Kryo v 2.24.0 will not handle (but 4.0.2 or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not ideal. I have two options left: go it alone and try to update Kryo in the apex-core, or simply change the default serializer. The hiccup I am running into is that I don't see how (from the documentation) to set the default serializer...perhaps I am missing it. Can someone point me to where that is in the docs?
On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> wrote: > Ah, you make a good point about the dev list...I am using IntelliJ on a > Mac. When I tried building without any changes, I also encountered > failures. I can re-run the non-modified sources to do a diff and see if > the errors were different... > > Sent from my iPhone > > On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.imman...@gmail.com> > wrote: > > Aaron, > > The question might be best served on the dev@apex.apache.org mailing list > as you are starting to make changes to the sources. Nothing obvious jumps > out to be, you don't need hadoop on your system to build the sources > successfully with all the tests passing. The test failures you are seeing > may or may not be related to the kryo change. What happens when you try to > build the original sources with changing the Kryo version? What is your > environment, mac, linux? > > On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> I know it has been a little while since I brought this up...I pulled down >> the 3.7.1-RELEASE branch from GitHub in order to test out the change to >> Kryo v. 4.0.2 and encountered no problems while building, however, when I >> try to do mvn install, there are several failed tests. I am hoping, that >> other than the obvious (to me, anyway) errors such as not finding >> yarn-site.xml on my laptop which does not have hadoop installed, someone >> may be able to see what is going on? >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramRecoveryTest >> >> ------------------------------------------------------------------------------- >> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec >> <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest >> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: >> 0.115 sec <<< FAILURE! >> java.lang.AssertionError: flush count expected:<1> but was:<2> >> at >> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) >> >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.engine.StatsTest >> >> ------------------------------------------------------------------------------- >> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 >> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest >> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) >> Time elapsed: 3.266 sec <<< FAILURE! >> java.lang.AssertionError: Validate input port queue size -1 >> at >> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) >> at >> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramMiniClusterTest >> >> ------------------------------------------------------------------------------- >> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec >> <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest >> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< >> ERROR! >> java.io.FileNotFoundException: >> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml >> (No such file or directory) >> at >> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramLocalClusterTest >> >> ------------------------------------------------------------------------------- >> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 >> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest >> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: >> 0.138 sec <<< FAILURE! >> java.lang.AssertionError: checkpoint path exists >> at >> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) >> >> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) >> Time elapsed: 5.191 sec <<< FAILURE! >> java.lang.AssertionError: >> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out >> exists >> at >> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.client.AppPackageTest >> >> ------------------------------------------------------------------------------- >> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec >> <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest >> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time >> elapsed: 0 sec <<< FAILURE! >> java.lang.AssertionError: Number of applications expected:<2> but was:<0> >> at >> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) >> >> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) >> Time elapsed: 0.001 sec <<< FAILURE! >> java.lang.AssertionError: Should consist of an app called >> MyFirstApplication >> at >> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) >> >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.cli.ApexCliTest >> >> ------------------------------------------------------------------------------- >> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec >> <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest >> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time >> elapsed: 0.104 sec <<< ERROR! >> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> Application Package >> at >> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) >> >> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.045 sec <<< FAILURE! >> org.junit.ComparisonFailure: expected:<[app]-default> but >> was:<[package]-default> >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) >> >> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.045 sec <<< ERROR! >> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> Application Package matching "MyFirstApplication" >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) >> >> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.058 sec <<< FAILURE! >> org.junit.ComparisonFailure: expected:<[app]-default> but >> was:<[package]-default> >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) >> >> >> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> Thanks, Thomas! I have implemented the fix you suggested >>> (@FieldSerializer.BInd) and also created a JIRA ticket: >>> https://issues.apache.org/jira/browse/APEXCORE-816 >>> >>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: >>> >>>> Hi, >>>> >>>> It is not necessary to use a different stream codec when you have >>>> control over the class that is causing the error. You can annotate a >>>> different field serializer instead, using the @FieldSerializer.Bind >>>> annotation. Here is an example: >>>> >>>> >>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >>>> >>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you >>>> like to create a JIRA for it? (http://apex.apache.org/contributing.html >>>> ) >>>> >>>> Thanks, >>>> Thomas >>>> >>>> >>>> >>>> >>>> >>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> >>>> wrote: >>>> >>>>> I am writing an application that includes a custom object. One of the >>>>> fields is of type Instant and seems to not agree with the version of Kryo >>>>> that is used in Apex (see exception below). I now that kryo 4.X supports >>>>> serializing Instant fields, but for the life of me, cannot seem to find a >>>>> way to either get Apex to use a newer version of Kryo. Can anyone help? >>>>> Perhaps setting a different streamcodec that uses the newer version of >>>>> kryo >>>>> would work, but I cannot seem to find an example of how to set a different >>>>> streamcodec despite references to the two (Kryo and Java) that ship with >>>>> Apex. >>>>> >>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>>>> Catastrophic Error: Execution halted due to Kryo exception! >>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created >>>>> (missing no-arg constructor): java.time.Instant >>>>> Serialization trace: >>>>> systemTime >>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >>>>> at >>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>> at com.datatorrent.stram.codec.De >>>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >>>>> at >>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >>>>> at >>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>>>> at >>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >>>>> >>>>> >>>>> -- >>>>> >>>>> M. Aaron Bossert >>>>> (571) 242-4021 >>>>> Punch Cyber Analytics Group >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> >>> M. Aaron Bossert >>> (571) 242-4021 >>> Punch Cyber Analytics Group >>> >>> >>> >> >> >> -- >> >> M. Aaron Bossert >> (571) 242-4021 >> Punch Cyber Analytics Group >> >> >> -- M. Aaron Bossert (571) 242-4021 Punch Cyber Analytics Group
