DanielLeens commented on issue #13314:
URL: https://github.com/apache/seatunnel/issues/13314#issuecomment-6983602245

   I reproduced the memory/IMap divergence and rejected deployment transition 
on `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5` with a focused Java 11 probe.
   
   Steps:
   1. Construct a real `PhysicalVertex` whose IMap state is `FAILED`.
   2. Make `IMap.set(location, CREATED)` throw 
`HazelcastInstanceNotActiveException`; leave reads returning the existing 
`FAILED` entry.
   3. Call the real `reset()`. The unchanged retry code performs **30 
attempts** over approximately **60 seconds**, logs the exhausted failure, then 
returns with in-memory state `CREATED` while the IMap still says `FAILED`.
   4. Call real `updateTaskState(DEPLOYING)`. The terminal-state guard reads 
`FAILED` from the IMap and rejects the transition; in-memory state stays 
`CREATED`.
   5. Control: with successful IMap writes, reset followed by the same 
transition reaches `DEPLOYING`.
   
   Observed output:
   ```text
   13314 REPRODUCED: retry calls=30, elapsedMs=60145, memory=CREATED, 
IMap=FAILED; DEPLOYING rejected
   13314 CONTROL: successful write permits DEPLOYING
   ```
   
   This confirms the vertex-level failure chain with actual retry exhaustion, 
not a live cluster's indefinitely stuck pipeline. The harness below also 
includes focused checks for #13323 and #13325, with separate scope notes.
   
   ### How I ran the focused probe
   
   Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK 
11.0.27. The production source is unmodified. These are **in-process 
fault-injection probes**, not end-to-end cluster/failover tests; passing the 
probe means its asserted failure signature was observed.
   
   Save the Java source below outside the checkout. From a clean checkout of 
that commit:
   
   ```bash
   export JAVA_HOME=/path/to/jdk-11
   export PATH="$JAVA_HOME/bin:$PATH"
   mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true package
   mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1
   CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 | 
sed 's/, /:/g')
   # The last reactor module is engine-server; CP uses these freshly compiled 
classes.
   ```
   
   This builds the required reactor modules, not the full project's test suite. 
No Docker or live database is needed. I also compared the tested revision with 
`dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes 
exercised here are unchanged between them. That comparison is not a separate 
runtime test of the newer commit.
   
   ```bash
   mkdir -p probe-classes
   javac -proc:none -cp "$CP" -d probe-classes StateRepros.java
   java -cp "probe-classes:$CP" 
org.apache.seatunnel.engine.server.dag.physical.StateRepros
   ```
   
   <details>
   <summary>Executable Java probe: StateRepros.java</summary>
   
   ```java
   /* Licensed to the Apache Software Foundation (ASF) under one or more
    * contributor license agreements. See the NOTICE file distributed with
    * this work for additional information regarding copyright ownership.
    * The ASF licenses this file to You under the Apache License, Version 2.0
    * (the "License"); you may not use this file except in compliance with
    * the License. You may obtain a copy at 
http://www.apache.org/licenses/LICENSE-2.0
    * Unless required by applicable law or agreed to in writing, software
    * distributed under the License is distributed on an "AS IS" BASIS,
    * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND. See the License for the
    * specific language governing permissions and limitations under the License.
    */
   package org.apache.seatunnel.engine.server.dag.physical;
   import org.apache.seatunnel.engine.common.Constant;
   import org.apache.seatunnel.engine.common.config.JobConfig;
   import org.apache.seatunnel.engine.core.job.JobImmutableInformation;
   import org.apache.seatunnel.engine.core.job.PipelineStatus;
   import org.apache.seatunnel.engine.server.execution.*;
   import org.apache.seatunnel.engine.server.master.JobMaster;
   import 
org.apache.seatunnel.engine.server.resourcemanager.resource.SlotProfile;
   import com.hazelcast.cluster.Address;
   import com.hazelcast.cluster.Member;
   import com.hazelcast.flakeidgen.FlakeIdGenerator;
   import com.hazelcast.map.IMap;
   import com.hazelcast.spi.impl.NodeEngine;
   import com.hazelcast.spi.impl.operationservice.impl.InvocationFuture;
   import com.hazelcast.core.HazelcastInstanceNotActiveException;
   import java.lang.reflect.Field;
   import java.util.*;
   import java.util.concurrent.*;
   import java.util.concurrent.atomic.*;
   import static org.mockito.Mockito.*;
   import static org.mockito.ArgumentMatchers.*;
   public class StateRepros {
    static void require(boolean x,String msg){if(!x)throw new 
AssertionError(msg);}
    static Object field(Object o,String n)throws Exception{Field 
f=o.getClass().getDeclaredField(n);f.setAccessible(true);return f.get(o);}
    static IMap<Object,Object> stateMap(Map<Object,Object> 
m){IMap<Object,Object> 
im=mock(IMap.class);when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
 null;}).when(im).set(any(),any());return im;}
    static IMap<Object,Long[]> times(){IMap<Object,Long[]> 
im=mock(IMap.class);Map<Object,Long[]> m=new 
HashMap<>();when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
 null;}).when(im).set(any(),any());return im;}
    static JobImmutableInformation info(){JobImmutableInformation 
i=mock(JobImmutableInformation.class);JobConfig cfg=new 
JobConfig();cfg.getEnvOptions().put("job.retry.times",3);when(i.getJobConfig()).thenReturn(cfg);when(i.getJobId()).thenReturn(1L);return
 i;}
    static PhysicalVertex vertex(IMap<Object,Object> map,NodeEngine 
node,TaskGroupLocation loc){TaskGroupDefaultImpl 
group=mock(TaskGroupDefaultImpl.class);when(group.getTaskGroupLocation()).thenReturn(loc);when(group.getTaskGroupName()).thenReturn("repro");return
 new 
PhysicalVertex(0,1,group,mock(FlakeIdGenerator.class),1,1,Collections.emptyList(),Collections.emptyList(),info(),System.currentTimeMillis(),node,map,times());}
    static void resetFailure()throws Exception{
     TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> 
m=new 
ConcurrentHashMap<>();m.put(loc,ExecutionState.FAILED);IMap<Object,Object> 
map=stateMap(m);
     PhysicalVertex v=vertex(map,mock(NodeEngine.class),loc);AtomicInteger 
calls=new AtomicInteger();
     doAnswer(a->{calls.incrementAndGet();throw new 
HazelcastInstanceNotActiveException("injected reset write 
failure");}).when(map).set(eq(loc),eq(ExecutionState.CREATED));
     long start=System.nanoTime();v.reset();
     require(v.getExecutionState()==ExecutionState.CREATED,"memory should 
reset");require(m.get(loc)==ExecutionState.FAILED,"durable stale state");
     
v.updateTaskState(ExecutionState.DEPLOYING);require(v.getExecutionState()==ExecutionState.CREATED,"deployment
 was rejected");
     System.out.println("13314 REPRODUCED: retry calls="+calls+", 
elapsedMs="+(System.nanoTime()-start)/1000000+", memory=CREATED, IMap=FAILED; 
DEPLOYING rejected");
     TaskGroupLocation ok=new TaskGroupLocation(2,1,1);Map<Object,Object> n=new 
HashMap<>();n.put(ok,ExecutionState.FAILED);PhysicalVertex 
control=vertex(stateMap(n),mock(NodeEngine.class),ok);control.reset();control.updateTaskState(ExecutionState.DEPLOYING);require(control.getExecutionState()==ExecutionState.DEPLOYING,"control
 transition");System.out.println("13314 CONTROL: successful write permits 
DEPLOYING");
    }
    static SubPlan plan(IMap<Object,Object> map,ExecutorService 
ex,List<PhysicalVertex> vs){return new 
SubPlan(1,1,System.currentTimeMillis(),vs,Collections.emptyList(),info(),ex,map,times(),Collections.emptyMap());}
    static void retryBudget()throws Exception{
     ExecutorService 
ex=Executors.newSingleThreadExecutor();try{Map<Object,Object> m=new 
HashMap<>();IMap<Object,Object> map=stateMap(m);JobMaster 
jm=mock(JobMaster.class);when(jm.isNeedRestore()).thenReturn(true);
     SubPlan 
p=plan(map,ex,Collections.emptyList());p.setJobMaster(jm);((AtomicInteger)field(p,"pipelineRestoreNum")).set(3);p.retainRestoreCause(3,"injected
 prior failure");require(!p.canRestorePipeline(),"old budget exhausted");
     SubPlan 
recovered=plan(map,ex,Collections.emptyList());recovered.setJobMaster(jm);require(recovered.getPipelineRestoreNum()==0,"reconstructed
 counter");require(recovered.canRestorePipeline(),"fresh 
budget");require(recovered.getRestoreCauses().isEmpty(),"history reset");
     System.out.println("13323 REPRODUCED: same IMap, old 
count=3/canRestore=false; reconstructed count=0/canRestore=true, causes empty. 
Constructor-level reconstruction, not multi-node failover.");
     }finally{ex.shutdownNow();}
    }
    static void cancelMonitors()throws Exception{
     ExecutorService ex=Executors.newSingleThreadExecutor();Thread 
worker=null,waiter=null;
     try{
      TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> 
m=new 
ConcurrentHashMap<>();m.put(loc,ExecutionState.RUNNING);IMap<Object,Object> 
map=stateMap(m);
      NodeEngine node=mock(NodeEngine.class,RETURNS_DEEP_STUBS);Address 
address=new Address("127.0.0.1",5801);com.hazelcast.cluster.impl.MemberImpl 
member=mock(com.hazelcast.cluster.impl.MemberImpl.class);when(member.getAddress()).thenReturn(address);when(node.getClusterService().getMembers()).thenReturn(Collections.singleton(member));when(node.getClusterService().getMember(address)).thenReturn(member);
      IMap owned=mock(IMap.class);SlotProfile 
slot=mock(SlotProfile.class);when(slot.getWorker()).thenReturn(address);when(owned.get(any())).thenReturn(Collections.singletonMap(loc,slot));when(node.getHazelcastInstance().getMap(Constant.IMAP_OWNED_SLOT_PROFILES)).thenReturn(owned);
      JobMaster 
jm=mock(JobMaster.class);when(jm.getOwnedSlotProfiles(loc)).thenReturn(slot);
      InvocationFuture<Object> 
probe=mock(InvocationFuture.class);when(probe.get()).thenReturn(true);
      InvocationFuture<Object> 
cancel=mock(InvocationFuture.class);AtomicInteger attempts=new 
AtomicInteger();CountDownLatch twice=new CountDownLatch(2);
      
when(cancel.get()).thenAnswer(a->{attempts.incrementAndGet();twice.countDown();throw
 new ExecutionException(new RuntimeException("injected non-responsive live 
worker"));});
      
when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CheckTaskGroupIsExecutingOperation.class),eq(address)).invoke()).thenReturn(probe);
      
when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CancelTaskOperation.class),eq(address)).invoke()).thenReturn(cancel);
      PhysicalVertex v=vertex(map,node,loc);v.setJobMaster(jm);
      SubPlan 
p=plan(map,ex,Collections.singletonList(v));p.setJobMaster(jm);p.isRunning=true;
      worker=new 
Thread(()->p.updatePipelineState(PipelineStatus.CANCELING),"cancel-state-thread");worker.start();require(twice.await(12,TimeUnit.SECONDS),"two
 retries expected");
      AtomicBoolean entered=new AtomicBoolean();waiter=new 
Thread(()->{synchronized(p){entered.set(true);}},"pipeline-monitor-waiter");waiter.start();Thread.sleep(250);
      require(waiter.getState()==Thread.State.BLOCKED&&!entered.get(),"pipeline 
monitor is held");
      
require(!((org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture<?>)field(v,"taskFuture")).isDone(),"cancel
 pending");
      System.out.println("13325 REPRODUCED: attempts="+attempts+", 
pipeline-monitor-waiter="+waiter.getState()+", cancel thread 
alive="+worker.isAlive());
      
when(node.getClusterService().getMember(address)).thenReturn(null);worker.join(5000);waiter.join(5000);require(!worker.isAlive()&&entered.get(),"member
 removal should unblock");
      System.out.println("13325 CONTROL: declaring worker absent releases 
monitors; vertex="+v.getExecutionState());
     } finally 
{if(worker!=null&&worker.isAlive())worker.interrupt();ex.shutdownNow();}
    }
    public static void main(String[] args)throws 
Exception{resetFailure();retryBudget();cancelMonitors();}
   }
   
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to