[ https://issues.apache.org/jira/browse/IGNITE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Zinoviev updated IGNITE-12658: ------------------------------------- Priority: Blocker (was: Critical) > [ML][Examples] TutorialStepByStepExample failed on cluster with more then 1 > node > -------------------------------------------------------------------------------- > > Key: IGNITE-12658 > URL: https://issues.apache.org/jira/browse/IGNITE-12658 > Project: Ignite > Issue Type: Bug > Components: examples, ml > Affects Versions: 2.8 > Environment: Ubuntu/Win > Java 8 > Reporter: Stepan Pilschikov > Assignee: Alexey Zinoviev > Priority: Blocker > > Steps to reproduce: > 1. Run Ignite node with org.apache.ignite.examples.ExampleNodeStartup (1 > node will be enough) > 2. Run org.apache.ignite.examples.ml.tutorial.TutorialStepByStepExample > Actual: > On Step_8_CV_with_Param_Grid_and_metrics starting to throw a lot of > exceptions > {code:java} > Train with p: 2 and maxDeep: 1 > >>> Trained model: if (x1 > 0.4368) then return 1.0000 else return 0.0000 > >>> Accuracy 0.7679083094555874 > >>> Test Error 0.2320916905444126 > >>> Tutorial step 8 (cross-validation) example completed. > [13:25:40] Ignite node stopped OK [uptime=00:00:17.453] > >>> Tutorial step 8 (cross-validation with param grid) example started. > [13:25:40] __________ ________________ > [13:25:40] / _/ ___/ |/ / _/_ __/ __/ > [13:25:40] _/ // (7 7 // / / / / _/ > [13:25:40] /___/\___/_/|_/___/ /_/ /___/ > [13:25:40] > [13:25:40] ver. 2.8.0#20200130-sha1:f478aa56 > [13:25:40] 2020 Copyright(C) Apache Software Foundation > [13:25:40] > [13:25:40] Ignite documentation: http://ignite.apache.org > [13:25:40] > [13:25:40] Quiet mode. > [13:25:40] ^-- Logging to file > '/opt/buildagent/work/d501ae8146bd8253/i2test/var/suite-examples/app-ignite/work/log/ignite-e156b2f2.log' > [13:25:40] ^-- Logging by 'Log4JLogger [quiet=true, config=null]' > [13:25:40] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or > "-v" to ignite.{sh|bat} > [13:25:40] > [13:25:40] OS: Linux 4.15.0-65-generic amd64 > [13:25:40] VM information: Java(TM) SE Runtime Environment 1.8.0_221-b11 > Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.221-b11 > [13:25:40] Please set system property '-Djava.net.preferIPv4Stack=true' to > avoid possible problems in mixed environments. > [13:25:40] Configured plugins: > [13:25:40] ^-- ml-inference-plugin 1.0.0 > [13:25:40] ^-- null > [13:25:40] > [13:25:40] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler > [tryStop=false, timeout=0, super=AbstractFailureHandler > [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] > [13:25:40] Message queue limit is set to 0 which may lead to potential OOMEs > when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to > message queues growth on sender and receiver sides. > [13:25:40] Security status [authentication=off, tls/ssl=off] > [13:25:41] Performance suggestions for grid (fix if possible) > [13:25:41] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > [13:25:41] ^-- Disable grid events (remove 'includeEventTypes' from > configuration) > [13:25:41] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM > options) > [13:25:41] ^-- Set max direct memory size if getting 'OOME: Direct buffer > memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) > [13:25:41] ^-- Disable processing of calls to System.gc() (add > '-XX:+DisableExplicitGC' to JVM options) > [13:25:41] Refer to this page for more performance suggestions: > https://apacheignite.readme.io/docs/jvm-and-system-tuning > [13:25:41] > [13:25:41] To start Console Management & Monitoring run > ignitevisorcmd.{sh|bat} > [13:25:41] Data Regions Configured: > [13:25:41] ^-- Default_Region [initSize=500.0 MiB, maxSize=18.9 GiB, > persistence=false, lazyMemoryAllocation=true] > [13:25:41] > [13:25:41] Ignite node started OK (id=e156b2f2) > [13:25:41] Topology snapshot [ver=20, locNode=e156b2f2, servers=2, clients=0, > state=ACTIVE, CPUs=5, offheap=38.0GB, heap=3.0GB] > [13:25:41] ^-- Baseline [id=0, size=2, online=2, offline=0] > [2020-02-11 13:25:42,428][ERROR][sys-#593][GridTaskWorker] Failed to obtain > remote job result policy for result from ComputeTask.result(..) method (will > fail the whole task): GridJobResultImpl [job=C2 > [c=o.a.i.ml.dataset.impl.cache.util.ComputeUtils$DeployableCallable@30e27659], > sib=GridJobSiblingImpl > [sesId=f9aced33071-e156b2f2-d116-4389-bd43-8536dc598888, > jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc598888, > nodeId=f1135598-73c8-4324-92af-45c110a6a637, isJobDone=false], > jobCtx=GridJobContextImpl > [jobId=1aaced33071-e156b2f2-d116-4389-bd43-8536dc598888, timeoutObj=null, > attrs=HashMap {}], node=TcpDiscoveryNode > [id=f1135598-73c8-4324-92af-45c110a6a637, > consistentId=127.0.0.1,192.168.112.2:47500, addrs=ArrayList [127.0.0.1, > 192.168.112.2], sockAddrs=HashSet [/127.0.0.1:47500, > 87bf29244900/192.168.112.2:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1581416741501, loc=false, ver=2.8.0#20200130-sha1:f478aa56, > isClient=false], ex=class o.a.i.IgniteException: Failed to deserialize object > [typeName=o.a.i.i.processors.closure.GridClosureProcessor$C2], hasRes=true, > isCancelled=false, isOccupied=true] > class org.apache.ignite.IgniteException: Remote job threw user exception > (override or implement ComputeTask.result(..) method if you would like to > have automatic failover for this exception): Failed to deserialize object > [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] > at > org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:102) > at > org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1062) > at > org.apache.ignite.internal.processors.task.GridTaskWorker$5.apply(GridTaskWorker.java:1055) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7037) > at > org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:1055) > at > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:862) > at > org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1146) > at > org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1370) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: class org.apache.ignite.IgniteException: Failed to deserialize > object > [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] > at > org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:467) > at > org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1265) > at > org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089) > ... 7 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > deserialize object > [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] > at > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10310) > at > org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:448) > ... 9 more > Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to > deserialize object > [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2] > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > at > org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:307) > at > org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:101) > at > org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:81) > at > org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10304) > ... 10 more > Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to > deserialize object > [typeName=org.apache.ignite.ml.dataset.impl.cache.util.ComputeUtils$DeployableCallable] > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > at > org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1800) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1331) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872) > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:864) > ... 16 more > Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to > read field [name=fun] > at > org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192) > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:874) > ... 22 more > Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to > deserialize object [typeName=java.lang.invoke.SerializedLambda] > at > org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:913) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) > at > org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984) > at > org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703) > at > org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188) > ... 23 more > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)