Hi folks, I wanted to share a new experimental project I've been working on and get your feedback and thoughts. As you know we are working on the modernization part of ManifoldCF, in the CONNECTORS-1784 branch you can find the upgrade to Java 25, Virtual Threads and Structured Concurrency. But the real problem, I mean in order to scale out in a modern way, is to separate the connectors from the framework. Basically we need to rewrite the connectors from scratch... At this point I'm wondering if we should address this modernization task in a more drastic way by rewriting ManifoldCF from scratch using modern frameworks.
We all know and appreciate the power of Apache ManifoldCF for enterprise document ingestion. However, as Java and the web ecosystem have evolved, I wanted to experiment with what a "next-generation" ingestion platform could look like if it were built from the ground up using modern technologies. The experiment is called Spring-Manifold Next-Gen and It is a proof-of-concept/skeleton that reimagines the crawler architecture using today's standard stack. Key Highlights: 1. Modern Java Concurrency: Built on JDK 25. Instead of custom thread pools, it uses Java Virtual Threads and Structured Concurrency to manage parallel repository scanning and document ingestion safely and efficiently. 2. Spring Boot Integration: Leverages Spring Boot for configuration, dependency injection, and management, replacing custom classloading and registry systems. 3. Vector & AI Pipelines out-of-the-box: Directly supports pgvector (PostgreSQL) and Ollama (local AI embeddings) to feed enterprise data into modern LLM/RAG search indices. 4. Modern Admin UI: Features a single-page application dashboard built with React and Vite. This is currently a skeleton with initial filesystem repository connector and vector output connector, but it proves the feasibility of using virtual threads for highly concurrent, scalable crawlers. I would love for you to check it out, share your feedback, and discuss whether this modern, Spring-aligned approach is something the community finds interesting or helpful for future directions. You can find the GitHub repository here: https://github.com/OpenPj/spring-manifold-next-gen Looking forward to your thoughts! Cheers, PG -- Piergiorgio
