[
https://issues.apache.org/jira/browse/IGNITE-29181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov reassigned IGNITE-29181:
----------------------------------------
Assignee: (was: Nikolay Izhikov)
> P2P: in SHARED mode a job class is requested from a neighbour client that
> does not have it (regression of IGNITE-20861)
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29181
> URL: https://issues.apache.org/jira/browse/IGNITE-29181
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.17
> Reporter: Nikolay Izhikov
> Priority: Major
> Labels: p2p
> Attachments: GridP2PClientDeploymentSourceTest.java
>
>
> Since IGNITE-20861 (2.17) a server in SHARED deployment mode can request a
> job class from a client node that does not have it, even though the client
> that submitted the task has it. The peer class loader gives up on the first
> node that reports the class missing, so the job fails.
> h3. Scenario
> Thick clients with partly overlapping class paths (different applications on
> each client):
> * Client A has task {{T1}} with job {{J1}}, and task {{T2}}.
> * Client B has task {{T2}} with its job, and may have {{T1}} but *not* {{J1}}.
> * Peer class loading enabled, SHARED mode (default).
> # A executes {{T2}}. The server creates a shared deployment {{D_A}} whose
> peer loader points at A.
> # B executes {{T2}}. In {{GridDeploymentPerVersionStore#getDeployment}} the
> block added by IGNITE-20861 scans every existing shared deployment and calls
> {{dep0.deployedClass(T2)}} on {{D_A}}. {{D_A}} loads {{T2}} from A, so B is
> registered as a participant of {{D_A}} with
> {{GridDeploymentClassLoader#register}}, which puts B *first* in the loader's
> node list.
> # A executes {{T1}}. {{GridDeploymentManager#getGlobalDeployment}} resolves
> A's request through {{searchDeploymentCache}} to {{D_A}} (A is its
> participant). {{D_A}}'s loader asks B first for {{T1}} and then for {{J1}}. B
> does not have {{J1}}, replies "not found", and
> {{GridDeploymentClassLoader#sendClassRequest}} breaks out of the node loop
> ("in case of shared resources/classes all nodes should have it") and caches
> the class as missed. The job fails with {{Failed to deserialize task
> attributes}} / {{P2PClassNotFoundException}}.
> In 2.16 every submitting node got a shared deployment of its own in this
> case, so {{D_A}}'s loader only ever asked A.
> Observed on a production cluster: "Ignite goes for the job code not to the
> initiator node, which has the classes, but to a neighbouring client node that
> has the task class but not the job class."
> h3. Reproducer
> {{GridP2PClientDeploymentSourceTest}} (attached, goes to
> {{org.apache.ignite.internal.managers.deployment}} because
> {{GridDeploymentRequest}} accessors are package-private; register it in
> {{IgniteP2PSelfTestSuite}}). It records every outgoing
> {{GridDeploymentRequest}} with its destination through
> {{TestRecordingCommunicationSpi}} and asserts that classes are requested only
> from nodes that hold them.
> * {{testNeighbourClientWithoutJobClass}}: one server, client A with all
> classes of {{ignite-extdata-p2p}}, client B whose loader excludes
> {{SingleSplitTestTask$SingleSplitTestJob}} and
> {{GridSingleSplitTestJobTarget}}. A runs {{P2PTestTaskExternalPath1}}, B runs
> {{P2PTestTaskExternalPath2}}, A runs {{SingleSplitTestTask}}.
> * {{testSharedMode}} / {{testContinuousMode}} / {{testPrivateMode}} /
> {{testIsolatedMode}}: two servers, two clients with the classes (separate
> loaders), two clients without; every request must target a client that has
> the classes, also after the first owner leaves.
> Bisect:
> ||commit||testNeighbourClientWithoutJobClass||other 4||
> |531eba2c86e (parent of IGNITE-20861)|pass|pass|
> |be246366346 IGNITE-20861|fail: {{SingleSplitTestJob}} requested from B|pass|
> |master|fail|pass|
> |master with the IGNITE-20861 block in {{getDeployment}} disabled|pass|pass|
> Log on the failing side:
> {noformat}
> Failed to find class on remote node
> [class=org.apache.ignite.tests.p2p.classic.SingleSplitTestTask$SingleSplitTestJob,
> nodeId=<client B>, clsLdrId=<client B loader>, ...]
> Remote job threw user exception ...: Failed to deserialize task attributes
> [taskName=org.apache.ignite.tests.p2p.classic.SingleSplitTestTask, ...,
> taskClsLdr=GridDeploymentClassLoader [..., nodeLdrMap=HashMap {<client
> B>=..., <client A>=...}, ...]]
> {noformat}
> h3. Possible fixes
> * Do not register a sender into a shared deployment it did not create just
> because that deployment can load the task class: a node holding the task
> class does not necessarily hold every class of the deployment.
> * Or make {{GridDeploymentClassLoader#sendClassRequest}} continue with the
> next participant when one reports the class missing instead of breaking and
> caching the miss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
