[
https://issues.apache.org/jira/browse/CONNECTORS-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093214#comment-18093214
]
Piergiorgio Lucidi commented on CONNECTORS-1784:
------------------------------------------------
I'm trying to summarize the latest changes applied to this branch:
h1. 1. Structured Concurrency ( StructuredTaskScope )
Structured concurrency is used to manage concurrent subtasks and scope
lifetimes safely (using try-with-resources):
• Socket Connections (InterruptibleSocketFactory.java):{{{}{}}}
{code:java}
Uses StructuredTaskScope with awaitAllSuccessfulOrThrow to fork socket
connection attempts:
try (var scope =
StructuredTaskScope.open(StructuredTaskScope.Joiner.<Socket>awaitAllSuccessfulOrThrow(),
...)) {
StructuredTaskScope.Subtask<Socket> subtask = scope.fork(() -> { ... });
...
}{code}
• User ACL Resolution (UserACLServlet.java):
Resolves ACL checks concurrently within a structured scope:
{code:java}
try (var scope = StructuredTaskScope.open()) { ... }{code}
h1. 2. Virtual Threads ( Thread.ofVirtual() )
All primary background processing threads, worker threads, and HTTP execution
threads have been migrated to Java's Virtual Threads:
• Crawler Engine Workers (CrawlerAgent.java):
All system threads, including Worker threads, Stuffer threads, Delete
threads, Cleanup threads, Finisher threads, and Seeding threads are created as
virtual threads:
{code:java}
workerThreads[i] = Thread.ofVirtual().name("Worker thread
'"+i+"'").unstarted(new WorkerThread(...));
stufferThread = Thread.ofVirtual().name("Stuffer thread").unstarted(new
StufferThread(...));{code}
• Agents Daemon (AgentsDaemon.java):
The central agents management threads run on virtual threads:
{code:java}
agentsThread = Thread.ofVirtual().name("Agents thread").unstarted(new
AgentsThread());{code}
• Outbound HTTP Client (CommonsHTTPSender.java):
Executes outbound HTTP request threads on virtual threads:
{code:java}
Thread vThread = Thread.ofVirtual().name("HTTP
request").start(methodThread);{code}
h1. 3. Jetty Thread Pool ( newVirtualThreadPerTaskExecutor )
Both the combined servlet runner and standalone runner have their task
executors configured to allocate a new Virtual Thread per incoming HTTP request:
• Jetty Combined Runner (ManifoldCFCombinedJettyRunner.java):
{code:java}
threadPool.setVirtualThreadsExecutor(Executors.newVirtualThreadPerTaskExecutor());{code}
• Jetty Standard Runner (ManifoldCFJettyRunner.java):
{code:java}
threadPool.setVirtualThreadsExecutor(Executors.newVirtualThreadPerTaskExecutor());{code}
> Core Performance & Modernization (The Java 21 Leap)
> ---------------------------------------------------
>
> Key: CONNECTORS-1784
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1784
> Project: ManifoldCF
> Issue Type: Task
> Components: Framework core
> Reporter: Piergiorgio Lucidi
> Assignee: Piergiorgio Lucidi
> Priority: Major
>
> The transition to OpenJDK 21 is the foundation for a more scalable and
> responsive architecture.
> *Virtual Threads Integration (Project Loom):*
> * Replace legacy thread pooling with *Virtual Threads* to handle thousands
> of concurrent repository connections with minimal memory overhead.
> * Implement *Structured Concurrency* to improve the reliability of complex
> crawling jobs and prevent resource leakage.
> *REST API v2 & OpenAPI Specification:*
> * Complete the transition to a fully documented RESTful API.
> * Enable "Configuration-as-Code" to support modern DevOps workflows and
> external orchestration.
> *Observability with OpenTelemetry:*
> * Integrate native tracing to monitor document processing latency across
> connectors, leveraging Java 21's improved profiling capabilities.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
