[
https://issues.apache.org/jira/browse/CONNECTORS-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083327#comment-18083327
]
Piergiorgio Lucidi edited comment on CONNECTORS-1784 at 5/25/26 4:22 PM:
-------------------------------------------------------------------------
Here are some of the main tasks currently implemented in the CONNECTORS-1784
branch:
* Migrated the build process to Maven: introduced a new distribution module
for generating all the needed resources replicating the Ant build
* Project upgraded to OpenJDK 25 in order to facilitate Virtual Threads and
Observability.
* Enabled Preview Features: Updated the Maven build (pom.xml) to include the
--enable-preview argument for the maven-compiler-plugin, maven-surefire-plugin,
and maven-failsafe-plugin, allowing the use of the StructuredTaskScope API.
* First implementation of Virtual Threads and Structured Concurrency.
* Jetty upgraded to version 12.1.8.
* Enabled JSP Support for Jetty 12 (EE 10).
* asm upgraded to version 9.10.
* Crawler Engine (CrawlerAgent): Replaced fixed-size arrays of platform
threads for Worker, Expire, Delete, and Cleanup pools with Virtual Threads
started via Thread.ofVirtual().
* Updated StructuredTaskScope usage to the Java 25 API, replacing
ShutdownOnFailure with the new StructuredTaskScope.open() and Joiner patterns.
* Refactored the per-query execution mechanism. Previously, ManifoldCF spawned
isolated ExecuteQueryThread instances (to bypass issues with drivers swallowing
interrupts). These are now executed within a
StructuredTaskScope.ShutdownOnFailure(), ensuring that any timeout or
interruption properly cleans up the database connection and avoids leaked query
threads.
* Refactor for the Authority Service (UserACLServlet): Parallel execution of
mapping and authorization checks now leverages Structured Concurrency. If an
error occurs while communicating with one authority, the structured scope
ensures all parallel constraint matches are safely cancelled and joined, rather
than leaving dangling threads in the servlet container.
* Refactor for the Web Crawler Connector (ThrottledFetcher): Replaced the
manually managed ExecuteMethodThread with a structured fork() call. This allows
the web crawler to reliably abort lingering HTTP connections to non-responsive
web servers without leaking resources during complex web snapshots.
* Refactored CmisRepositoryConnector and CmisOutputConnector to execute their
session acquisition, connection checks, and teardown routines inside a
StructuredTaskScope.
* Updated ModifiedCloudSolrClient to use Structured Concurrency for parallel
shard updates, replacing the legacy ThreadPoolExecutor and Future.get() loops.
If one replica fails during an update, the scope manages the failure cleanly
and prevents orphaned update tasks from consuming resources.
was (Author: [email protected]):
Here are some of the main tasks currently implemented in the CONNECTORS-1784
branch:
* Project upgraded to OpenJDK 25 in order to facilitate Virtual Threads and
Observability.
* Enabled Preview Features: Updated the Maven build (pom.xml) to include the
--enable-preview argument for the maven-compiler-plugin, maven-surefire-plugin,
and maven-failsafe-plugin, allowing the use of the StructuredTaskScope API.
* First implementation of Virtual Threads and Structured Concurrency.
* Jetty upgraded to version 12.1.8.
* Enabled JSP Support for Jetty 12 (EE 10).
* asm upgraded to version 9.10.
* Crawler Engine (CrawlerAgent): Replaced fixed-size arrays of platform
threads for Worker, Expire, Delete, and Cleanup pools with Virtual Threads
started via Thread.ofVirtual().
* Updated StructuredTaskScope usage to the Java 25 API, replacing
ShutdownOnFailure with the new StructuredTaskScope.open() and Joiner patterns.
* Refactored the per-query execution mechanism. Previously, ManifoldCF spawned
isolated ExecuteQueryThread instances (to bypass issues with drivers swallowing
interrupts). These are now executed within a
StructuredTaskScope.ShutdownOnFailure(), ensuring that any timeout or
interruption properly cleans up the database connection and avoids leaked query
threads.
* Refactor for the Authority Service (UserACLServlet): Parallel execution of
mapping and authorization checks now leverages Structured Concurrency. If an
error occurs while communicating with one authority, the structured scope
ensures all parallel constraint matches are safely cancelled and joined, rather
than leaving dangling threads in the servlet container.
* Refactor for the Web Crawler Connector (ThrottledFetcher): Replaced the
manually managed ExecuteMethodThread with a structured fork() call. This allows
the web crawler to reliably abort lingering HTTP connections to non-responsive
web servers without leaking resources during complex web snapshots.
* Refactored CmisRepositoryConnector and CmisOutputConnector to execute their
session acquisition, connection checks, and teardown routines inside a
StructuredTaskScope.
* Updated ModifiedCloudSolrClient to use Structured Concurrency for parallel
shard updates, replacing the legacy ThreadPoolExecutor and Future.get() loops.
If one replica fails during an update, the scope manages the failure cleanly
and prevents orphaned update tasks from consuming resources.
> Core Performance & Modernization (The Java 21 Leap)
> ---------------------------------------------------
>
> Key: CONNECTORS-1784
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1784
> Project: ManifoldCF
> Issue Type: Task
> Components: Framework core
> Reporter: Piergiorgio Lucidi
> Assignee: Piergiorgio Lucidi
> Priority: Major
>
> The transition to OpenJDK 21 is the foundation for a more scalable and
> responsive architecture.
> *Virtual Threads Integration (Project Loom):*
> * Replace legacy thread pooling with *Virtual Threads* to handle thousands
> of concurrent repository connections with minimal memory overhead.
> * Implement *Structured Concurrency* to improve the reliability of complex
> crawling jobs and prevent resource leakage.
> *REST API v2 & OpenAPI Specification:*
> * Complete the transition to a fully documented RESTful API.
> * Enable "Configuration-as-Code" to support modern DevOps workflows and
> external orchestration.
> *Observability with OpenTelemetry:*
> * Integrate native tracing to monitor document processing latency across
> connectors, leveraging Java 21's improved profiling capabilities.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
