DanielLeens commented on issue #13314:
URL: https://github.com/apache/seatunnel/issues/13314#issuecomment-6983602245
I reproduced the memory/IMap divergence and rejected deployment transition
on `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5` with a focused Java 11 probe.
Steps:
1. Construct a real `PhysicalVertex` whose IMap state is `FAILED`.
2. Make `IMap.set(location, CREATED)` throw
`HazelcastInstanceNotActiveException`; leave reads returning the existing
`FAILED` entry.
3. Call the real `reset()`. The unchanged retry code performs **30
attempts** over approximately **60 seconds**, logs the exhausted failure, then
returns with in-memory state `CREATED` while the IMap still says `FAILED`.
4. Call real `updateTaskState(DEPLOYING)`. The terminal-state guard reads
`FAILED` from the IMap and rejects the transition; in-memory state stays
`CREATED`.
5. Control: with successful IMap writes, reset followed by the same
transition reaches `DEPLOYING`.
Observed output:
```text
13314 REPRODUCED: retry calls=30, elapsedMs=60145, memory=CREATED,
IMap=FAILED; DEPLOYING rejected
13314 CONTROL: successful write permits DEPLOYING
```
This confirms the vertex-level failure chain with actual retry exhaustion,
not a live cluster's indefinitely stuck pipeline. The harness below also
includes focused checks for #13323 and #13325, with separate scope notes.
### How I ran the focused probe
Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK
11.0.27. The production source is unmodified. These are **in-process
fault-injection probes**, not end-to-end cluster/failover tests; passing the
probe means its asserted failure signature was observed.
Save the Java source below outside the checkout. From a clean checkout of
that commit:
```bash
export JAVA_HOME=/path/to/jdk-11
export PATH="$JAVA_HOME/bin:$PATH"
mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \
-DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
-Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
-Dmaven.javadoc.skip=true -Drat.skip=true package
mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \
-DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
-Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
-Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1
CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 |
sed 's/, /:/g')
# The last reactor module is engine-server; CP uses these freshly compiled
classes.
```
This builds the required reactor modules, not the full project's test suite.
No Docker or live database is needed. I also compared the tested revision with
`dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes
exercised here are unchanged between them. That comparison is not a separate
runtime test of the newer commit.
```bash
mkdir -p probe-classes
javac -proc:none -cp "$CP" -d probe-classes StateRepros.java
java -cp "probe-classes:$CP"
org.apache.seatunnel.engine.server.dag.physical.StateRepros
```
<details>
<summary>Executable Java probe: StateRepros.java</summary>
```java
/* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy at
http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND. See the License for the
* specific language governing permissions and limitations under the License.
*/
package org.apache.seatunnel.engine.server.dag.physical;
import org.apache.seatunnel.engine.common.Constant;
import org.apache.seatunnel.engine.common.config.JobConfig;
import org.apache.seatunnel.engine.core.job.JobImmutableInformation;
import org.apache.seatunnel.engine.core.job.PipelineStatus;
import org.apache.seatunnel.engine.server.execution.*;
import org.apache.seatunnel.engine.server.master.JobMaster;
import
org.apache.seatunnel.engine.server.resourcemanager.resource.SlotProfile;
import com.hazelcast.cluster.Address;
import com.hazelcast.cluster.Member;
import com.hazelcast.flakeidgen.FlakeIdGenerator;
import com.hazelcast.map.IMap;
import com.hazelcast.spi.impl.NodeEngine;
import com.hazelcast.spi.impl.operationservice.impl.InvocationFuture;
import com.hazelcast.core.HazelcastInstanceNotActiveException;
import java.lang.reflect.Field;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import static org.mockito.Mockito.*;
import static org.mockito.ArgumentMatchers.*;
public class StateRepros {
static void require(boolean x,String msg){if(!x)throw new
AssertionError(msg);}
static Object field(Object o,String n)throws Exception{Field
f=o.getClass().getDeclaredField(n);f.setAccessible(true);return f.get(o);}
static IMap<Object,Object> stateMap(Map<Object,Object>
m){IMap<Object,Object>
im=mock(IMap.class);when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
null;}).when(im).set(any(),any());return im;}
static IMap<Object,Long[]> times(){IMap<Object,Long[]>
im=mock(IMap.class);Map<Object,Long[]> m=new
HashMap<>();when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
null;}).when(im).set(any(),any());return im;}
static JobImmutableInformation info(){JobImmutableInformation
i=mock(JobImmutableInformation.class);JobConfig cfg=new
JobConfig();cfg.getEnvOptions().put("job.retry.times",3);when(i.getJobConfig()).thenReturn(cfg);when(i.getJobId()).thenReturn(1L);return
i;}
static PhysicalVertex vertex(IMap<Object,Object> map,NodeEngine
node,TaskGroupLocation loc){TaskGroupDefaultImpl
group=mock(TaskGroupDefaultImpl.class);when(group.getTaskGroupLocation()).thenReturn(loc);when(group.getTaskGroupName()).thenReturn("repro");return
new
PhysicalVertex(0,1,group,mock(FlakeIdGenerator.class),1,1,Collections.emptyList(),Collections.emptyList(),info(),System.currentTimeMillis(),node,map,times());}
static void resetFailure()throws Exception{
TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object>
m=new
ConcurrentHashMap<>();m.put(loc,ExecutionState.FAILED);IMap<Object,Object>
map=stateMap(m);
PhysicalVertex v=vertex(map,mock(NodeEngine.class),loc);AtomicInteger
calls=new AtomicInteger();
doAnswer(a->{calls.incrementAndGet();throw new
HazelcastInstanceNotActiveException("injected reset write
failure");}).when(map).set(eq(loc),eq(ExecutionState.CREATED));
long start=System.nanoTime();v.reset();
require(v.getExecutionState()==ExecutionState.CREATED,"memory should
reset");require(m.get(loc)==ExecutionState.FAILED,"durable stale state");
v.updateTaskState(ExecutionState.DEPLOYING);require(v.getExecutionState()==ExecutionState.CREATED,"deployment
was rejected");
System.out.println("13314 REPRODUCED: retry calls="+calls+",
elapsedMs="+(System.nanoTime()-start)/1000000+", memory=CREATED, IMap=FAILED;
DEPLOYING rejected");
TaskGroupLocation ok=new TaskGroupLocation(2,1,1);Map<Object,Object> n=new
HashMap<>();n.put(ok,ExecutionState.FAILED);PhysicalVertex
control=vertex(stateMap(n),mock(NodeEngine.class),ok);control.reset();control.updateTaskState(ExecutionState.DEPLOYING);require(control.getExecutionState()==ExecutionState.DEPLOYING,"control
transition");System.out.println("13314 CONTROL: successful write permits
DEPLOYING");
}
static SubPlan plan(IMap<Object,Object> map,ExecutorService
ex,List<PhysicalVertex> vs){return new
SubPlan(1,1,System.currentTimeMillis(),vs,Collections.emptyList(),info(),ex,map,times(),Collections.emptyMap());}
static void retryBudget()throws Exception{
ExecutorService
ex=Executors.newSingleThreadExecutor();try{Map<Object,Object> m=new
HashMap<>();IMap<Object,Object> map=stateMap(m);JobMaster
jm=mock(JobMaster.class);when(jm.isNeedRestore()).thenReturn(true);
SubPlan
p=plan(map,ex,Collections.emptyList());p.setJobMaster(jm);((AtomicInteger)field(p,"pipelineRestoreNum")).set(3);p.retainRestoreCause(3,"injected
prior failure");require(!p.canRestorePipeline(),"old budget exhausted");
SubPlan
recovered=plan(map,ex,Collections.emptyList());recovered.setJobMaster(jm);require(recovered.getPipelineRestoreNum()==0,"reconstructed
counter");require(recovered.canRestorePipeline(),"fresh
budget");require(recovered.getRestoreCauses().isEmpty(),"history reset");
System.out.println("13323 REPRODUCED: same IMap, old
count=3/canRestore=false; reconstructed count=0/canRestore=true, causes empty.
Constructor-level reconstruction, not multi-node failover.");
}finally{ex.shutdownNow();}
}
static void cancelMonitors()throws Exception{
ExecutorService ex=Executors.newSingleThreadExecutor();Thread
worker=null,waiter=null;
try{
TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object>
m=new
ConcurrentHashMap<>();m.put(loc,ExecutionState.RUNNING);IMap<Object,Object>
map=stateMap(m);
NodeEngine node=mock(NodeEngine.class,RETURNS_DEEP_STUBS);Address
address=new Address("127.0.0.1",5801);com.hazelcast.cluster.impl.MemberImpl
member=mock(com.hazelcast.cluster.impl.MemberImpl.class);when(member.getAddress()).thenReturn(address);when(node.getClusterService().getMembers()).thenReturn(Collections.singleton(member));when(node.getClusterService().getMember(address)).thenReturn(member);
IMap owned=mock(IMap.class);SlotProfile
slot=mock(SlotProfile.class);when(slot.getWorker()).thenReturn(address);when(owned.get(any())).thenReturn(Collections.singletonMap(loc,slot));when(node.getHazelcastInstance().getMap(Constant.IMAP_OWNED_SLOT_PROFILES)).thenReturn(owned);
JobMaster
jm=mock(JobMaster.class);when(jm.getOwnedSlotProfiles(loc)).thenReturn(slot);
InvocationFuture<Object>
probe=mock(InvocationFuture.class);when(probe.get()).thenReturn(true);
InvocationFuture<Object>
cancel=mock(InvocationFuture.class);AtomicInteger attempts=new
AtomicInteger();CountDownLatch twice=new CountDownLatch(2);
when(cancel.get()).thenAnswer(a->{attempts.incrementAndGet();twice.countDown();throw
new ExecutionException(new RuntimeException("injected non-responsive live
worker"));});
when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CheckTaskGroupIsExecutingOperation.class),eq(address)).invoke()).thenReturn(probe);
when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CancelTaskOperation.class),eq(address)).invoke()).thenReturn(cancel);
PhysicalVertex v=vertex(map,node,loc);v.setJobMaster(jm);
SubPlan
p=plan(map,ex,Collections.singletonList(v));p.setJobMaster(jm);p.isRunning=true;
worker=new
Thread(()->p.updatePipelineState(PipelineStatus.CANCELING),"cancel-state-thread");worker.start();require(twice.await(12,TimeUnit.SECONDS),"two
retries expected");
AtomicBoolean entered=new AtomicBoolean();waiter=new
Thread(()->{synchronized(p){entered.set(true);}},"pipeline-monitor-waiter");waiter.start();Thread.sleep(250);
require(waiter.getState()==Thread.State.BLOCKED&&!entered.get(),"pipeline
monitor is held");
require(!((org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture<?>)field(v,"taskFuture")).isDone(),"cancel
pending");
System.out.println("13325 REPRODUCED: attempts="+attempts+",
pipeline-monitor-waiter="+waiter.getState()+", cancel thread
alive="+worker.isAlive());
when(node.getClusterService().getMember(address)).thenReturn(null);worker.join(5000);waiter.join(5000);require(!worker.isAlive()&&entered.get(),"member
removal should unblock");
System.out.println("13325 CONTROL: declaring worker absent releases
monitors; vertex="+v.getExecutionState());
} finally
{if(worker!=null&&worker.isAlive())worker.interrupt();ex.shutdownNow();}
}
public static void main(String[] args)throws
Exception{resetFailure();retryBudget();cancelMonitors();}
}
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]