Ah, you make a good point about the dev list...I am using IntelliJ on a Mac. When I tried building without any changes, I also encountered failures. I can re-run the non-modified sources to do a diff and see if the errors were different...
Sent from my iPhone > On Jun 11, 2018, at 15:11, Pramod Immaneni <pramod.imman...@gmail.com> wrote: > > Aaron, > > The question might be best served on the d...@apex.apache.org mailing list as > you are starting to make changes to the sources. Nothing obvious jumps out to > be, you don't need hadoop on your system to build the sources successfully > with all the tests passing. The test failures you are seeing may or may not > be related to the kryo change. What happens when you try to build the > original sources with changing the Kryo version? What is your environment, > mac, linux? > >> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> wrote: >> I know it has been a little while since I brought this up...I pulled down >> the 3.7.1-RELEASE branch from GitHub in order to test out the change to Kryo >> v. 4.0.2 and encountered no problems while building, however, when I try to >> do mvn install, there are several failed tests. I am hoping, that other >> than the obvious (to me, anyway) errors such as not finding yarn-site.xml on >> my laptop which does not have hadoop installed, someone may be able to see >> what is going on? >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramRecoveryTest >> ------------------------------------------------------------------------------- >> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec >> <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest >> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: >> 0.115 sec <<< FAILURE! >> java.lang.AssertionError: flush count expected:<1> but was:<2> >> at >> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.engine.StatsTest >> ------------------------------------------------------------------------------- >> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec >> <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest >> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) >> Time elapsed: 3.266 sec <<< FAILURE! >> java.lang.AssertionError: Validate input port queue size -1 >> at >> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) >> at >> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramMiniClusterTest >> ------------------------------------------------------------------------------- >> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec >> <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest >> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< >> ERROR! >> java.io.FileNotFoundException: >> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml >> (No such file or directory) >> at >> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.StramLocalClusterTest >> ------------------------------------------------------------------------------- >> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec >> <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest >> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: >> 0.138 sec <<< FAILURE! >> java.lang.AssertionError: checkpoint path exists >> at >> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) >> >> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) >> Time elapsed: 5.191 sec <<< FAILURE! >> java.lang.AssertionError: >> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out >> exists >> at >> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.client.AppPackageTest >> ------------------------------------------------------------------------------- >> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec >> <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest >> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed: 0 >> sec <<< FAILURE! >> java.lang.AssertionError: Number of applications expected:<2> but was:<0> >> at >> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) >> >> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) >> Time elapsed: 0.001 sec <<< FAILURE! >> java.lang.AssertionError: Should consist of an app called MyFirstApplication >> at >> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) >> >> >> ------------------------------------------------------------------------------- >> Test set: com.datatorrent.stram.cli.ApexCliTest >> ------------------------------------------------------------------------------- >> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec >> <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest >> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time >> elapsed: 0.104 sec <<< ERROR! >> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> Application Package >> at >> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) >> >> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.045 sec <<< FAILURE! >> org.junit.ComparisonFailure: expected:<[app]-default> but >> was:<[package]-default> >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) >> >> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.045 sec <<< ERROR! >> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> Application Package matching "MyFirstApplication" >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) >> >> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> Time elapsed: 0.058 sec <<< FAILURE! >> org.junit.ComparisonFailure: expected:<[app]-default> but >> was:<[package]-default> >> at >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) >> >> >>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> wrote: >>> Thanks, Thomas! I have implemented the fix you suggested >>> (@FieldSerializer.BInd) and also created a JIRA ticket: >>> https://issues.apache.org/jira/browse/APEXCORE-816 >>> >>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: >>>> Hi, >>>> >>>> It is not necessary to use a different stream codec when you have control >>>> over the class that is causing the error. You can annotate a different >>>> field serializer instead, using the @FieldSerializer.Bind annotation. Here >>>> is an example: >>>> >>>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >>>> >>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you >>>> like to create a JIRA for it? (http://apex.apache.org/contributing.html) >>>> >>>> Thanks, >>>> Thomas >>>> >>>> >>>> >>>> >>>> >>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> >>>>> wrote: >>>>> I am writing an application that includes a custom object. One of the >>>>> fields is of type Instant and seems to not agree with the version of Kryo >>>>> that is used in Apex (see exception below). I now that kryo 4.X supports >>>>> serializing Instant fields, but for the life of me, cannot seem to find a >>>>> way to either get Apex to use a newer version of Kryo. Can anyone help? >>>>> Perhaps setting a different streamcodec that uses the newer version of >>>>> kryo would work, but I cannot seem to find an example of how to set a >>>>> different streamcodec despite references to the two (Kryo and Java) that >>>>> ship with Apex. >>>>> >>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>>>> Catastrophic Error: Execution halted due to Kryo exception! >>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing >>>>> no-arg constructor): java.time.Instant >>>>> Serialization trace: >>>>> systemTime >>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >>>>> at >>>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >>>>> at >>>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>> at >>>>> com.datatorrent.stram.codec.DefaultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >>>>> at >>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >>>>> at >>>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >>>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>>>> at >>>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >>>>> >>>>> >>>>> -- >>>>> >>>>> M. Aaron Bossert >>>>> (571) 242-4021 >>>>> Punch Cyber Analytics Group >>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> >>> M. Aaron Bossert >>> (571) 242-4021 >>> Punch Cyber Analytics Group >>> >>> >> >> >> >> -- >> >> M. Aaron Bossert >> (571) 242-4021 >> Punch Cyber Analytics Group >> >>
