Hi PJ, Thank you for the feedback and for highlighting Apache StormCrawler.
I completely agree that exploring collaboration between the two projects is a great idea. While StormCrawler is an incredibly powerful tool for large scale web crawling and processing, OpenCrawling was built specifically to tackle the enterprise content and process automation ecosystem. Our core focus is bridging platforms like Alfresco, Flowable, and Camunda directly into modern LLM and RAG architectures, heavily utilizing Spring Boot and Spring AI. Despite the distinct use cases, web data versus enterprise repositories, there is absolutely a shared interest in robust data ingestion, document parsing and vector database integration strategies. OpenCrawling can become the perfect home for any crawling strategy. I would be thrilled to connect with the StormCrawler community to see how our projects might complement each other and share best practices moving forward. We could propose to implement a brand new OpenCrawling Storm Bolt. In Apache Storm topology, data flows from *Spouts* (URL queues) to *Bolts* (fetchers, parsers and indexers). The most seamless integration is to build an opencrawling-storm-bolt. StormCrawler handles the heavy lifting of recursive web crawling, politeness and HTML parsing. Instead of using StormCrawler's native OpenSearch/Elasticsearch indexer bolt, the topology passes the parsed document to the new custom OpenCrawling Storm Bolt. The Bolt acts as an OpenCrawling Repository Connector. It takes the raw HTML, wraps it in the Open Ingestion Standard (OIS) format, attaches any relevant baseline metadata and pushes it through OpenCrawling’s secure pipeline (Java 25 / Spring Boot 4) for chunking, embedding, and Vector DB ingestion. Cheers, PG Il Mar 11 Ago 2026, 18:44 PJ Fanning <[email protected]> ha scritto: > This doesn't block OpenCrawling joining as an ASF Incubator podling > but I just want to highlight that there is already Apache > StormCrawler. > > https://stormcrawler.apache.org/ > > It would be great if these projects could collaborate in areas of > shared interest. > > On Tue, 11 Aug 2026 at 15:28, Piergiorgio Lucidi <[email protected]> > wrote: > > > > Hi everyone, > > > > I would like to propose OpenCrawling as a new project for incubation > within > > the Apache Software Foundation. Currently the project is hosted on GitHub > > [1]. > > > > The official proposal is currently available in the OpenCrawling Wiki in > > markdown format [2]. > > > > I tried to share the proposal in our Confluence but it seems that I don't > > have permission to create the new page under the Proposals page. Anyway > if > > someone can guide me on resolving this issue it would be great! > > > > Once space permissions are granted on cwiki.apache.org, I will also > mirror > > the proposal on the Incubator CWIKI proposals page. > > Below you also find the same proposal ready to be copy-pasted into our > > Confluence. > > > > We welcome feedback, questions and discussion from the Incubator > community! > > > > Best regards, > > Piergiorgio > > On behalf of the OpenCrawling Core Team > > > > [1] - https://github.com/opencrawling/opencrawling > > [2] - > > > https://github.com/opencrawling/opencrawling/wiki/Apache-Incubator-Proposal > > > > -------------------------------------------------------------- > > > > h1. Apache Incubator Proposal: OpenCrawling > > > > h2. Abstract > > > > *OpenCrawling* is an open-source, enterprise-grade, high-performance data > > crawling, content ingestion, and security-aware vector search platform. > > Built on modern Java 25 (leveraging Virtual Threads and Structured > > Concurrency), Spring Boot 4, and Spring AI, OpenCrawling serves as the > > reference implementation of the *Open Ingestion Standard (OIS)* and > > provides a secure *Model Context Protocol (MCP)* server interface. It > > orchestrates scalable data flows from heterogeneous enterprise > repositories > > (e.g., SharePoint, S3, CMIS/Alfresco, BPMN engines like Camunda and > > Flowable, Apache Iceberg, Apache Ozone) to downstream vector databases > > (e.g., Milvus, Qdrant, OpenSearch, Vespa, pgvector) with source-level > > Access Control List (ACL) security enforcement. > > > > ---- > > > > h2. Proposal > > > > The OpenCrawling community proposes to incubate *OpenCrawling* as a new > > project within the Apache Software Foundation (ASF). OpenCrawling > provides > > a decoupled, vendor-neutral enterprise data integration framework that > > bridges the gap between traditional enterprise content management (ECM) > > repositories and modern Large Language Model (LLM) / Retrieval-Augmented > > Generation (RAG) architectures. > > > > The project encompasses: > > # *Core Ingestion Runtime ({{oc-core}}, {{oc-runtime}})*: A distributed, > > asynchronous engine built with virtual threads, claim-check metadata > > patterns, and Apache Kafka event streams. > > # *Repository & Vector Connectors*: Standardized connectors for scanning > > source systems and indexing vector embeddings into major vector stores. > > # *Open Ingestion Standard (OIS)*: The formal JSON/YAML schema > > specifications defining unified document payloads, ACL security SIDs, and > > crawler job configurations. > > # *Secure Model Context Protocol (MCP) Server*: A Zero-Trust context > > retrieval server enforcing document-level permissions (Active Directory > > SIDs, LDAP groups, user principals) at query time. > > # *Observability & Developer Tooling*: AI-Powered Observability (AIOps) > > over OpenTelemetry traces, Auto-Narrativization Copilot, Java Client SDK, > > and Maven Archetypes for custom connector development. > > > > ---- > > > > h2. Background > > > > In enterprise AI and RAG architectures, LLM agents require seamless > access > > to unstructured content stored across legacy and cloud repositories. > > However, traditional ingestion pipelines often strip out or ignore > > source-level security metadata (ACLs), leading to context leakage where > an > > AI model synthesizes responses using confidential documents that the > > requesting user does not have permissions to view. > > > > Furthermore, legacy crawling tools (such as Apache ManifoldCF or Apache > > Nutch) were architected over a decade ago prior to the emergence of > vector > > databases, LLMs, Model Context Protocol (MCP), and modern Java features > > like Virtual Threads (JEP 444) and Structured Concurrency. > > > > OpenCrawling was created to address this modern ingestion crisis by > > providing a native Java 25/Spring AI implementation engineered > specifically > > for LLM search scenarios, zero-trust context retrieval, and > high-throughput > > asynchronous processing. > > > > ---- > > > > h2. Rationale > > > > The Apache Software Foundation is the natural home for OpenCrawling. ASF > > has long been the center of innovation for enterprise search and big data > > infrastructure, hosting cornerstone projects such as Apache Lucene, > Apache > > Solr, Apache Tika, Apache Kafka, Apache Iceberg, Apache Ozone, Apache > > ManifoldCF, and Apache Nutch. > > > > Bringing OpenCrawling to the ASF offers multiple mutual benefits: > > * *Ecosystem Integration*: OpenCrawling directly integrates with and > builds > > upon existing Apache projects, including *Apache Tika* (text extraction), > > *Apache Kafka* (event-driven pipeline), *Apache Iceberg* (lakehouse > > connector), *Apache Ozone* (claim-check object storage), and *Apache > Maven* > > (connector archetype distribution). > > * *Vendor-Neutral Governance*: Neutral governance under the Apache Way is > > vital to establishing OpenCrawling and OIS as industry-wide, > > vendor-agnostic ingestion standards. > > * *Community Sustainability*: Operating as an Apache project will > attract a > > broader community of enterprise adopters, cloud providers, AI framework > > developers, and search engine vendors. > > > > ---- > > > > h2. Initial Goals > > > > During incubation, the OpenCrawling project will focus on the following > > milestones: > > > > # *ASF Migration & Infrastructure*: > > ** Transfer codebases ({{opencrawling}}, {{open-ingestion-standard}}, {{ > > opencrawling.github.io}}) to Apache infrastructure ({{ > > github.com/apache/incubator-opencrawling}} > <http://github.com/apache/incubator-opencrawling%7D%7D>). > > ** Rebrand build artifacts to {{org.apache.opencrawling}}. > > ** Setup ASF-compliant CI/CD pipelines using GitHub Actions. > > # *Community & Governance*: > > ** Adopt the Apache Way for all decisions, roadmap discussions, and > release > > voting. > > ** Expand the contributor base across independent developers, enterprise > > search users, and corporate contributors. > > # *Ecosystem & Connector Expansion*: > > ** Release additional output connectors (Elasticsearch, Apache Solr, > > RESTHeart). > > ** Add native integration for fine-grained authorization frameworks > (e.g., > > OpenFGA). > > ** Enhance gRPC support for high-efficiency inter-microservice > > communication. > > ** Standardize OIS specification drafts under ASF governance. > > # *Compliance & Licensing*: > > ** Complete IP clearance and execute software grant agreements. > > ** Ensure all third-party dependencies strictly conform to Apache License > > Category A policies. > > > > ---- > > > > h2. Current Status > > > > h3. Meritocracy > > The OpenCrawling project was established with meritocratic principles > from > > day one. Design decisions, architecture changes, issue tracking, and > > roadmap discussions take place openly on GitHub through RFCs, Pull > > Requests, and public wiki pages. > > > > h3. Community > > The OpenCrawling community includes developers and architects from > > enterprise search, ECM, and AI background. Community channels include > > GitHub Discussions, Slack, and social media announcements. The project > > actively encourages external contributions via Maven archetypes and > modular > > connector development. > > > > h3. Core Developers > > The initial core developers are experienced software architects and > > open-source veterans with extensive experience in enterprise search, > > content management, and ASF governance: > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Founder, Lead > > Architect. ASF Member and PMC Member/Committer on multiple Apache > projects > > (including Apache ManifoldCF and Apache Chemistry). > > * *Michael Cizmar* ({{[email protected]}}) – Lead Architect & > > Developer. Specialist in enterprise search and cloud infrastructure. > > * *Luis Cabaceira* ({{[email protected]}}) – Lead Architect & > > Developer. Specialist in document processing and AI integration. > > > > ---- > > > > h2. Known Risks > > > > h3. Orphaned Products > > The risk of OpenCrawling becoming orphaned is low. The project solves an > > active, urgent security and performance problem in enterprise AI adoption > > (RAG ACL context leakage). The core maintainers are committed to its > > long-term evolution and actively use it in production environments. > > > > h3. Inexperience with Open Source > > The project leadership has deep experience with open-source communities. > > Piergiorgio Lucidi is an active ASF Member and PMC member with over a > > decade of experience guiding projects through the Apache Way. > > > > h3. Homogenous Developers > > The initial committers come from diverse geographical locations (Italy, > > United States, Portugal) and distinct organizations/consultancies. > > Incubating at Apache will further diversify the developer base by > > encouraging contributions from enterprise organizations and search > vendors. > > > > h3. Reliance on Third-Party Products > > OpenCrawling is designed to be vendor-neutral. Core dependencies are > > open-source libraries under permissive licenses (Apache 2.0, MIT, BSD): > > * Spring Boot & Spring AI (Apache 2.0) > > * Apache Tika (Apache 2.0) > > * Apache Kafka (Apache 2.0) > > * PostgreSQL / pgvector (PostgreSQL License / MIT) > > * Docker & OpenTelemetry (Apache 2.0) > > > > There are no GPL/AGPL dependencies in the runtime core. > > > > h3. Relationship with Sponsored Products / Brand > > OpenCrawling is an independent project. The name "OpenCrawling" has been > > used for the open-source codebase. The trademark will be transferred to > the > > Apache Software Foundation upon incubation acceptance. > > > > ---- > > > > h2. Documentation & Existing Artifacts > > > > * *GitHub Organization*: [https://github.com/opencrawling] > > * *Main Code Base*: {{opencrawling/opencrawling}} > > * *Specification Repo*: {{opencrawling/open-ingestion-standard}} > > * *Documentation & Wiki*: [ > https://github.com/opencrawling/opencrawling/wiki > > ] > > * *Java Client SDK*: {{oc-java-client-sdk}} ([Sonatype Central - > > org.opencrawling:oc-java-client-sdk| > > > https://central.sonatype.com/artifact/org.opencrawling/oc-java-client-sdk > ]) > > * *Maven Archetypes*: [Sonatype Central - org.opencrawling.archetypes| > > > https://central.sonatype.com/artifact/org.opencrawling.archetypes/opencrawling-connector-archetypes > > ] > > > > ---- > > > > h2. Initial Source & Intellectual Property Submission > > > > h3. Initial Source Code > > The initial codebase to be granted to the ASF resides in the following > > GitHub repositories: > > * {{opencrawling/opencrawling}} (Core engine, microservices, UI, > > connectors, MCP server) > > * {{opencrawling/open-ingestion-standard}} (JSON schemas, whitepaper, > > specifications) > > * {{opencrawling/opencrawling.github.io}} (Project web site and > > documentation source) > > > > All source code is currently licensed under the *Apache License, Version > > 2.0*. > > > > h3. Software Grant / ICLA / CCLA > > All core contributors will submit Individual Contributor License > Agreements > > (ICLAs) and corporate software grants will be executed upon acceptance > into > > the Incubator. > > > > ---- > > > > h2. External Dependencies > > > > All major external dependencies of OpenCrawling use Apache-compatible > > licenses (Category A): > > > > || Dependency || License || > > | *Java Development Kit (JDK 25)* | GPLv2 + Classpath Exception | > > | *Spring Boot / Spring AI* | Apache License 2.0 | > > | *Apache Tika* | Apache License 2.0 | > > | *Apache Kafka Clients* | Apache License 2.0 | > > | *Apache Iceberg SDK* | Apache License 2.0 | > > | *Apache Ozone Client* | Apache License 2.0 | > > | *Jackson / Slf4j / Logback* | Apache 2.0 / MIT / EPL 1.0 | > > | *Milvus / Qdrant Java SDKs* | Apache License 2.0 | > > | *OpenTelemetry Java SDK* | Apache License 2.0 | > > | *React / Vite / Tailwind (Admin UI)* | MIT | > > > > ---- > > > > h2. Cryptography > > > > OpenCrawling uses standard TLS/HTTPS protocols and hashing routines > > provided by the standard Java Virtual Machine (JDK) and Spring Security > > framework for secure transport. It does not include custom cryptographic > > algorithms or controlled export software. > > > > ---- > > > > h2. Required Resources > > > > h3. Mailing Lists > > * {{[email protected]}} > > * {{[email protected]}} > > * {{[email protected]}} (PPMC) > > > > h3. Git Repositories > > * {{https://github.com/apache/incubator-opencrawling}} > > * {{https://github.com/apache/incubator-opencrawling-site}} > > > > h3. Issue Tracking > > * GitHub Issues on {{apache/incubator-opencrawling}} (or ASF Jira project > > {{OPENCRAWLING}}) > > > > h3. CI/CD Infrastructure > > * GitHub Actions workflows for automated build, test, multi-arch Docker > > image generation, and Sonar/Scorecard quality checks. > > > > ---- > > > > h2. Initial Committers & PPMC Members > > > > * *Piergiorgio Lucidi* ({{[email protected]}}) – Initial Committer > & > > PPMC > > * *Michael Cizmar* ({{[email protected]}}) – Initial Committer & > > PPMC > > * *Luis Cabaceira* ({{[email protected]}}) – Initial Committer & > PPMC > > > > _(Note: Additional mentors and committers will be welcomed during the > > discussion period on {{[email protected]}}.)_ > > > > ---- > > > > h2. Champions & Mentors > > > > * *Champion*: Piergiorgio Lucidi ({{[email protected]}}) – ASF > Member > > * *Mentors*: > > ** _(TBD - Interested ASF Members/Incubator PMC members invited to step > > forward during proposal discussion)_ > > > > ---- > > > > h2. Sponsoring Entity > > > > The *Apache Incubator PMC* is requested to be the sponsoring entity for > > this project. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > Piergiorgio Lucidi Mobile: 3395381669
