I know it has been a little while since I brought this up...I pulled down the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo v. 4.0.2 and encountered no problems while building, however, when I try to do mvn install, there are several failed tests. I am hoping, that other than the obvious (to me, anyway) errors such as not finding yarn-site.xml on my laptop which does not have hadoop installed, someone may be able to see what is going on?
------------------------------------------------------------------------------- Test set: com.datatorrent.stram.StramRecoveryTest ------------------------------------------------------------------------------- Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: 0.115 sec <<< FAILURE! java.lang.AssertionError: flush count expected:<1> but was:<2> at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) ------------------------------------------------------------------------------- Test set: com.datatorrent.stram.engine.StatsTest ------------------------------------------------------------------------------- Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) Time elapsed: 3.266 sec <<< FAILURE! java.lang.AssertionError: Validate input port queue size -1 at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) ------------------------------------------------------------------------------- Test set: com.datatorrent.stram.StramMiniClusterTest ------------------------------------------------------------------------------- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< ERROR! java.io.FileNotFoundException: /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml (No such file or directory) at com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) ------------------------------------------------------------------------------- Test set: com.datatorrent.stram.StramLocalClusterTest ------------------------------------------------------------------------------- Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: 0.138 sec <<< FAILURE! java.lang.AssertionError: checkpoint path exists at com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: 5.191 sec <<< FAILURE! java.lang.AssertionError: ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out exists at com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) ------------------------------------------------------------------------------- Test set: com.datatorrent.stram.client.AppPackageTest ------------------------------------------------------------------------------- Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed: 0 sec <<< FAILURE! java.lang.AssertionError: Number of applications expected:<2> but was:<0> at com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) Time elapsed: 0.001 sec <<< FAILURE! java.lang.AssertionError: Should consist of an app called MyFirstApplication at com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) ------------------------------------------------------------------------------- Test set: com.datatorrent.stram.cli.ApexCliTest ------------------------------------------------------------------------------- Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.104 sec <<< ERROR! com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package at com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.045 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default> at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.045 sec <<< ERROR! com.datatorrent.stram.cli.ApexCli$CliException: No applications in Application Package matching "MyFirstApplication" at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time elapsed: 0.058 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<[app]-default> but was:<[package]-default> at com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> wrote: > Thanks, Thomas! I have implemented the fix you suggested > (@FieldSerializer.BInd) and also created a JIRA ticket: > https://issues.apache.org/jira/browse/APEXCORE-816 > > On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: > >> Hi, >> >> It is not necessary to use a different stream codec when you have control >> over the class that is causing the error. You can annotate a different >> field serializer instead, using the @FieldSerializer.Bind annotation. Here >> is an example: >> >> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7 >> fd361a97549b6e7e12d4289/library/src/main/java/org/ >> apache/apex/malhar/lib/window/WindowOption.java#L50 >> >> It would be good to upgrade Kryo to 4.x in apex-core master. Would you >> like to create a JIRA for it? (http://apex.apache.org/contributing.html) >> >> Thanks, >> Thomas >> >> >> >> >> >> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> I am writing an application that includes a custom object. One of the >>> fields is of type Instant and seems to not agree with the version of Kryo >>> that is used in Apex (see exception below). I now that kryo 4.X supports >>> serializing Instant fields, but for the life of me, cannot seem to find a >>> way to either get Apex to use a newer version of Kryo. Can anyone help? >>> Perhaps setting a different streamcodec that uses the newer version of kryo >>> would work, but I cannot seem to find an example of how to set a different >>> streamcodec despite references to the two (Kryo and Java) that ship with >>> Apex. >>> >>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>> Catastrophic Error: Execution halted due to Kryo exception! >>> com.esotericsoftware.kryo.KryoException: Class cannot be created >>> (missing no-arg constructor): java.time.Instant >>> Serialization trace: >>> systemTime (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxE >>> vent) >>> at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.n >>> ewInstantiatorOf(Kryo.java:1228) >>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>> at com.esotericsoftware.kryo.serializers.FieldSerializer.create >>> (FieldSerializer.java:547) >>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(F >>> ieldSerializer.java:523) >>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>> at com.esotericsoftware.kryo.serializers.ObjectField.read(Objec >>> tField.java:113) >>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(F >>> ieldSerializer.java:528) >>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>> at com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromD >>> ataStatePair(DefaultStatefulStreamCodec.java:98) >>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferRe >>> servoir.processPayload(BufferServerSubscriber.java:391) >>> at com.datatorrent.stram.stream.BufferServerSubscriber$BufferRe >>> servoir.sweep(BufferServerSubscriber.java:339) >>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>> at com.datatorrent.stram.engine.StreamingContainer$2.run(Stream >>> ingContainer.java:1429) >>> >>> >>> -- >>> >>> M. Aaron Bossert >>> (571) 242-4021 >>> Punch Cyber Analytics Group >>> >>> >>> >> > > > -- > > M. Aaron Bossert > (571) 242-4021 > Punch Cyber Analytics Group > > > -- M. Aaron Bossert (571) 242-4021 Punch Cyber Analytics Group
