Nikolay Izhikov created IGNITE-29181:
----------------------------------------
Summary: P2P: in SHARED mode a job class is requested from a
neighbour client that does not have it (regression of IGNITE-20861)
Key: IGNITE-29181
URL: https://issues.apache.org/jira/browse/IGNITE-29181
Project: Ignite
Issue Type: Bug
Affects Versions: 2.17
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
Attachments: GridP2PClientDeploymentSourceTest.java
Since IGNITE-20861 (2.17) a server in SHARED deployment mode can request a job
class from a client node that does not have it, even though the client that
submitted the task has it. The peer class loader gives up on the first node
that reports the class missing, so the job fails.
h3. Scenario
Thick clients with partly overlapping class paths (different applications on
each client):
* Client A has task {{T1}} with job {{J1}}, and task {{T2}}.
* Client B has task {{T2}} with its job, and may have {{T1}} but *not* {{J1}}.
* Peer class loading enabled, SHARED mode (default).
# A executes {{T2}}. The server creates a shared deployment {{D_A}} whose peer
loader points at A.
# B executes {{T2}}. In {{GridDeploymentPerVersionStore#getDeployment}} the
block added by IGNITE-20861 scans every existing shared deployment and calls
{{dep0.deployedClass(T2)}} on {{D_A}}. {{D_A}} loads {{T2}} from A, so B is
registered as a participant of {{D_A}} with
{{GridDeploymentClassLoader#register}}, which puts B *first* in the loader's
node list.
# A executes {{T1}}. {{GridDeploymentManager#getGlobalDeployment}} resolves A's
request through {{searchDeploymentCache}} to {{D_A}} (A is its participant).
{{D_A}}'s loader asks B first for {{T1}} and then for {{J1}}. B does not have
{{J1}}, replies "not found", and {{GridDeploymentClassLoader#sendClassRequest}}
breaks out of the node loop ("in case of shared resources/classes all nodes
should have it") and caches the class as missed. The job fails with {{Failed to
deserialize task attributes}} / {{P2PClassNotFoundException}}.
In 2.16 every submitting node got a shared deployment of its own in this case,
so {{D_A}}'s loader only ever asked A.
Observed on a production cluster: "Ignite goes for the job code not to the
initiator node, which has the classes, but to a neighbouring client node that
has the task class but not the job class."
h3. Reproducer
{{GridP2PClientDeploymentSourceTest}} (attached, goes to
{{org.apache.ignite.internal.managers.deployment}} because
{{GridDeploymentRequest}} accessors are package-private; register it in
{{IgniteP2PSelfTestSuite}}). It records every outgoing
{{GridDeploymentRequest}} with its destination through
{{TestRecordingCommunicationSpi}} and asserts that classes are requested only
from nodes that hold them.
* {{testNeighbourClientWithoutJobClass}}: one server, client A with all classes
of {{ignite-extdata-p2p}}, client B whose loader excludes
{{SingleSplitTestTask$SingleSplitTestJob}} and
{{GridSingleSplitTestJobTarget}}. A runs {{P2PTestTaskExternalPath1}}, B runs
{{P2PTestTaskExternalPath2}}, A runs {{SingleSplitTestTask}}.
* {{testSharedMode}} / {{testContinuousMode}} / {{testPrivateMode}} /
{{testIsolatedMode}}: two servers, two clients with the classes (separate
loaders), two clients without; every request must target a client that has the
classes, also after the first owner leaves.
Bisect:
||commit||testNeighbourClientWithoutJobClass||other 4||
|531eba2c86e (parent of IGNITE-20861)|pass|pass|
|be246366346 IGNITE-20861|fail: {{SingleSplitTestJob}} requested from B|pass|
|master|fail|pass|
|master with the IGNITE-20861 block in {{getDeployment}} disabled|pass|pass|
Log on the failing side:
{noformat}
Failed to find class on remote node
[class=org.apache.ignite.tests.p2p.classic.SingleSplitTestTask$SingleSplitTestJob,
nodeId=<client B>, clsLdrId=<client B loader>, ...]
Remote job threw user exception ...: Failed to deserialize task attributes
[taskName=org.apache.ignite.tests.p2p.classic.SingleSplitTestTask, ...,
taskClsLdr=GridDeploymentClassLoader [..., nodeLdrMap=HashMap {<client B>=...,
<client A>=...}, ...]]
{noformat}
h3. Possible fixes
* Do not register a sender into a shared deployment it did not create just
because that deployment can load the task class: a node holding the task class
does not necessarily hold every class of the deployment.
* Or make {{GridDeploymentClassLoader#sendClassRequest}} continue with the next
participant when one reports the class missing instead of breaking and caching
the miss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
