This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 1963d4755813 CAMEL-26227: camel-lucene - index the fields that insert
writes, one document per exchange (#28529)
1963d4755813 is described below
commit 1963d4755813cc8f5ad104ab06a8d8655962e253
Author: Vignesh Nayak Manel <[email protected]>
AuthorDate: Wed Oct 28 11:40:28 2026 +0530
CAMEL-26227: camel-lucene - index the fields that insert writes, one
document per exchange (#28529)
* CAMEL-26227: camel-lucene - index the fields that insert writes, one
document per exchange
The Lucene 5 upgrade (CAMEL-9360) commented out FieldType.setIndexed(true)
and did not replace it with setIndexOptions, so the fields written by the
insert endpoint and by the srcDir indexing kept IndexOptions.NONE: they
were stored but never indexed, and a query never found them.
Each field was also written as a document of its own, so a hit on the
body could not be tied to the headers of its exchange or the path of its
file. One exchange, or one source file, is now one document.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* CAMEL-26227: camel-lucene - a header named contents does not replace the
body in the hit
The hit data is Document.get("contents"), which returns the first value of
the field. With one document per exchange, a header named contents was
added before the body and so became the hit data. Add exchangeId and
contents before the headers.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* CAMEL-26227: camel-lucene - a failed srcDir insert releases the index
write lock
On an index written by an older Camel, addDocument of a srcDir file throws
IllegalArgumentException (a field cannot go from index options NONE to
indexed).
The index writer was then never closed, and Lucene keeps its write lock
held in
the JVM, so every later IndexWriter on that indexDir failed with
LockObtainFailedException. Roll the writer back on failure, as index() does.
The component doc now says that srcDir files in subdirectories are indexed
too.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* CAMEL-26227: camel-lucene - indexing srcDir again replaces the documents
of its files
The files of srcDir are indexed each time the insert endpoint is created,
into
an index opened with CREATE_OR_APPEND, so every application start added
another
copy of each file. The copies could not be seen while nothing was indexed;
now
a query on a word of a file gave one hit per start. Replace the document
with
the same path (an indexed, untokenized field) instead of adding one.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../camel/catalog/docs/lucene-component.adoc | 8 +-
.../src/main/docs/lucene-component.adoc | 8 +-
.../camel/component/lucene/LuceneIndexer.java | 41 ++++--
.../lucene/LuceneIndexProducerLifecycleTest.java | 52 +++++++
.../component/lucene/LuceneInsertAndQueryTest.java | 161 +++++++++++++++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 19 +++
6 files changed, 275 insertions(+), 14 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/lucene-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/lucene-component.adoc
index bb73af022fd3..f4b88396aae8 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/lucene-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/lucene-component.adoc
@@ -61,8 +61,12 @@ include::partial$component-endpoint-headers.adoc[]
This component supports 2 producer endpoints.
*insert*: the insert producer builds a searchable index by analyzing
-the body in incoming exchanges and associating it with a token
-("content").
+the body in incoming exchanges and associating it with the field
+`contents`. Each exchange is indexed as one document, which holds the
+body in the field `contents`, the exchange id in the field `exchangeId`,
+and each header in a field of the header name. Each file under the `srcDir`
+directory, including its subdirectories, is indexed as one document, which
holds the file contents in
+the field `contents` and the file path, not tokenized, in the field `path`.
*query*: the query producer performs searches on a pre-created index.
The query uses the searchable index to perform score & relevance based
searches. Queries are sent via the incoming exchange contains a header
diff --git a/components/camel-lucene/src/main/docs/lucene-component.adoc
b/components/camel-lucene/src/main/docs/lucene-component.adoc
index bb73af022fd3..f4b88396aae8 100644
--- a/components/camel-lucene/src/main/docs/lucene-component.adoc
+++ b/components/camel-lucene/src/main/docs/lucene-component.adoc
@@ -61,8 +61,12 @@ include::partial$component-endpoint-headers.adoc[]
This component supports 2 producer endpoints.
*insert*: the insert producer builds a searchable index by analyzing
-the body in incoming exchanges and associating it with a token
-("content").
+the body in incoming exchanges and associating it with the field
+`contents`. Each exchange is indexed as one document, which holds the
+body in the field `contents`, the exchange id in the field `exchangeId`,
+and each header in a field of the header name. Each file under the `srcDir`
+directory, including its subdirectories, is indexed as one document, which
holds the file contents in
+the field `contents` and the file path, not tokenized, in the field `path`.
*query*: the query producer performs searches on a pre-created index.
The query uses the searchable index to perform score & relevance based
searches. Queries are sent via the incoming exchange contains a header
diff --git
a/components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneIndexer.java
b/components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneIndexer.java
index 4234cb8141d2..b02b64239c94 100644
---
a/components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneIndexer.java
+++
b/components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneIndexer.java
@@ -30,9 +30,11 @@ import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
+import org.apache.lucene.index.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.index.Term;
import org.apache.lucene.store.NIOFSDirectory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -75,18 +77,23 @@ public class LuceneIndexer {
LOG.debug("Indexing {}", exchange);
openIndexWriter();
try {
+ // one document per exchange, so a hit on the body or on a header
gives all the fields of its exchange
+ Document doc = new Document();
+ // before the headers: Document.get returns the first value of a
field, so a header named
+ // exchangeId or contents does not replace them in the hit
+ add(doc, "exchangeId", exchange.getExchangeId(), true);
+ add(doc, "contents",
exchange.getIn().getMandatoryBody(String.class), true);
Map<String, Object> headers = exchange.getIn().getHeaders();
- add("exchangeId", exchange.getExchangeId(), true);
for (Entry<String, Object> entry : headers.entrySet()) {
String field = entry.getKey();
String value =
exchange.getContext().getTypeConverter().mandatoryConvertTo(String.class,
entry.getValue());
- add(field, value, true);
+ add(doc, field, value, true);
}
- add("contents", exchange.getIn().getMandatoryBody(String.class),
true);
+ indexWriter.addDocument(doc);
indexWriter.commit();
} catch (Exception e) {
- // discard the documents of this exchange and release the index
write lock
+ // discard the document of this exchange and release the index
write lock
try {
indexWriter.rollback();
} catch (Exception re) {
@@ -121,15 +128,13 @@ public class LuceneIndexer {
this.analyzer = analyzer;
}
- private void add(String field, String value, boolean analyzed) throws
IOException {
+ private static void add(Document doc, String field, String value, boolean
analyzed) {
if (LOG.isTraceEnabled()) {
LOG.trace("Adding field: {}", field);
LOG.trace(" value: {}", value);
}
- Document doc = new Document();
doc.add(new Field(field, value, analyzed ? TOKENIZED_FIELD_TYPE :
NON_TOKENIZED_FIELD_TYPE));
- indexWriter.addDocument(doc);
}
private void add(File file) throws IOException {
@@ -151,10 +156,24 @@ public class LuceneIndexer {
IOHelper.copy(IOHelper.buffered(is), bos);
contents = bos.toString();
}
+ // one document per file, so a hit on the contents gives the
path of its file
+ Document doc = new Document();
+ add(doc, "path", file.getPath(), false);
+ add(doc, "contents", contents, true);
openIndexWriter();
- add("path", file.getPath(), false);
- add("contents", contents, true);
- closeIndexWriter();
+ try {
+ // replace the document of this path: srcDir is indexed
again each time the endpoint is created
+ indexWriter.updateDocument(new Term("path",
file.getPath()), doc);
+ closeIndexWriter();
+ } catch (Exception e) {
+ // discard the document of this file and release the index
write lock
+ try {
+ indexWriter.rollback();
+ } catch (Exception re) {
+ e.addSuppressed(re);
+ }
+ throw e;
+ }
LOG.trace("Added {} successfully", file);
}
@@ -179,6 +198,8 @@ public class LuceneIndexer {
private static FieldType createFieldType(boolean tokenized) {
FieldType answer = new FieldType();
+ // a FieldType is not indexed unless its index options are set (the
default is IndexOptions.NONE)
+ answer.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS);
answer.setStored(true);
answer.setTokenized(tokenized);
diff --git
a/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneIndexProducerLifecycleTest.java
b/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneIndexProducerLifecycleTest.java
index 5ef5a1991651..7e33f9c8a2a5 100644
---
a/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneIndexProducerLifecycleTest.java
+++
b/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneIndexProducerLifecycleTest.java
@@ -18,6 +18,8 @@ package org.apache.camel.component.lucene;
import java.io.File;
import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
import java.util.Collection;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
@@ -29,8 +31,15 @@ import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.spi.Registry;
import org.apache.camel.test.junit6.CamelTestSupport;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.StoredField;
import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.StoredFields;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.TermQuery;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.NIOFSDirectory;
import org.junit.jupiter.api.Test;
@@ -41,6 +50,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
@@ -132,6 +142,48 @@ class LuceneIndexProducerLifecycleTest extends
CamelTestSupport {
assertEquals(1, countIndexedMessages());
}
+ @Test
+ void failedSourceDirectoryInsertDoesNotHoldTheLock(@TempDir File
oldIndexDir, @TempDir File srcDir) throws Exception {
+ Files.writeString(new File(srcDir, "quote.txt").toPath(), "A day
without sunshine is like night",
+ StandardCharsets.UTF_8);
+ // an index written by an older Camel: contents is stored but not
indexed
+ try (FSDirectory dir = FSDirectory.open(oldIndexDir.toPath());
+ IndexWriter writer = new IndexWriter(dir, new
IndexWriterConfig(new StandardAnalyzer()))) {
+ Document doc = new Document();
+ doc.add(new StoredField("contents", "old"));
+ writer.addDocument(doc);
+ }
+
+ IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> new LuceneIndexer(srcDir, oldIndexDir, new
StandardAnalyzer()));
+ assertTrue(e.getMessage().contains("from index options=NONE"),
e.getMessage());
+
+ try (FSDirectory dir = FSDirectory.open(oldIndexDir.toPath());
+ IndexWriter writer = new IndexWriter(dir, new
IndexWriterConfig(new StandardAnalyzer()))) {
+ assertTrue(writer.isOpen(), "a failed source directory insert must
not leave the index write lock held");
+ }
+ }
+
+ @Test
+ void sourceDirectoryIndexedAgainReplacesTheDocumentsOfItsFiles(@TempDir
File srcIndexDir, @TempDir File srcDir)
+ throws Exception {
+ File file = new File(srcDir, "quote.txt");
+ Files.writeString(file.toPath(), "A day without sunshine is like
night", StandardCharsets.UTF_8);
+ new LuceneIndexer(srcDir, srcIndexDir, new StandardAnalyzer());
+
+ // as when the application starts again, with a file that was changed
in the meantime
+ Files.writeString(file.toPath(), "A day without rain is like night",
StandardCharsets.UTF_8);
+ new LuceneIndexer(srcDir, srcIndexDir, new StandardAnalyzer());
+
+ try (FSDirectory dir = FSDirectory.open(srcIndexDir.toPath());
+ DirectoryReader reader = DirectoryReader.open(dir)) {
+ assertEquals(1, reader.numDocs(), "indexing the source directory
again must not add a copy of its file");
+ IndexSearcher searcher = new IndexSearcher(reader);
+ assertEquals(1, searcher.count(new TermQuery(new Term("contents",
"rain"))), "the new contents");
+ assertEquals(0, searcher.count(new TermQuery(new Term("contents",
"sunshine"))), "the old contents");
+ }
+ }
+
@Test
void concurrentInsertsAreIndexed() throws Exception {
CountDownLatch writerOpen = new CountDownLatch(1);
diff --git
a/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneInsertAndQueryTest.java
b/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneInsertAndQueryTest.java
new file mode 100644
index 000000000000..721668b433cc
--- /dev/null
+++
b/components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneInsertAndQueryTest.java
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.lucene;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.processor.lucene.support.Hit;
+import org.apache.camel.processor.lucene.support.Hits;
+import org.apache.camel.spi.Registry;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.store.FSDirectory;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * What the insert endpoint writes must be found by the query endpoint, and
the fields of one exchange (or one source
+ * file) must be one document.
+ */
+class LuceneInsertAndQueryTest extends CamelTestSupport {
+
+ @TempDir
+ File indexDir;
+
+ @TempDir
+ File srcIndexDir;
+
+ @TempDir
+ File srcDir;
+
+ @Override
+ protected void bindToRegistry(Registry registry) throws Exception {
+ Files.writeString(new File(srcDir, "quote.txt").toPath(), "A day
without sunshine is like night",
+ StandardCharsets.UTF_8);
+ registry.bind("indexDir", indexDir);
+ registry.bind("srcIndexDir", srcIndexDir);
+ registry.bind("srcDir", srcDir);
+ registry.bind("stdAnalyzer", new StandardAnalyzer());
+ }
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+
from("direct:insert").to("lucene:insertIndex:insert?analyzer=#stdAnalyzer&indexDir=#indexDir");
+
from("direct:query").to("lucene:queryIndex:query?analyzer=#stdAnalyzer&indexDir=#indexDir&maxHits=20");
+ from("direct:insertWithSrcDir")
+
.to("lucene:srcIndex:insert?analyzer=#stdAnalyzer&indexDir=#srcIndexDir&srcDir=#srcDir");
+ from("direct:querySrcDir")
+
.to("lucene:srcQuery:query?analyzer=#stdAnalyzer&indexDir=#srcIndexDir&maxHits=20");
+ }
+ };
+ }
+
+ private Hits query(String endpoint, String phrase) {
+ Exchange exchange = template.send(endpoint, e -> {
+ e.getIn().setHeader(LuceneConstants.HEADER_QUERY, phrase);
+ e.getIn().setHeader(LuceneConstants.HEADER_RETURN_LUCENE_DOCS,
"true");
+ });
+ assertNull(exchange.getException(), "query " + phrase);
+ return exchange.getMessage().getBody(Hits.class);
+ }
+
+ @Test
+ void insertedBodyIsFoundByQuery() {
+ Exchange inserted = template.send("direct:insert", e ->
e.getIn().setBody("the quick brown fox"));
+ assertNull(inserted.getException(), "insert");
+
+ Hits hits = query("direct:query", "fox");
+ assertEquals(1, hits.getNumberOfHits(), "a word of the inserted body
must be found");
+ assertEquals("the quick brown fox", hits.getHit().get(0).getData());
+ }
+
+ @Test
+ void headersAndBodyOfOneExchangeAreOneDocument() throws Exception {
+ Exchange inserted = template.send("direct:insert", e -> {
+ e.getIn().setHeader("author", "carlin");
+ e.getIn().setBody("I think, therefore I am");
+ });
+ assertNull(inserted.getException(), "insert");
+
+ try (FSDirectory dir = FSDirectory.open(indexDir.toPath());
+ DirectoryReader reader = DirectoryReader.open(dir)) {
+ assertEquals(1, reader.numDocs(), "one exchange must be one
document");
+ }
+
+ Hits hits = query("direct:query", "author:carlin");
+ assertEquals(1, hits.getNumberOfHits(), "a header of the exchange must
be found");
+ Hit hit = hits.getHit().get(0);
+ assertEquals("I think, therefore I am", hit.getData(), "a hit on a
header must carry the body of its exchange");
+ Document document = hit.getDocument();
+ assertEquals("carlin", document.get("author"));
+ assertEquals(inserted.getExchangeId(), document.get("exchangeId"));
+ }
+
+ @Test
+ void headersNamedLikeTheExchangeFieldsDoNotReplaceThem() {
+ Exchange inserted = template.send("direct:insert", e -> {
+ e.getIn().setHeader("contents", "header value");
+ e.getIn().setHeader("exchangeId", "header id");
+ e.getIn().setBody("the real body");
+ });
+ assertNull(inserted.getException(), "insert");
+
+ Hits hits = query("direct:query", "real");
+ assertEquals(1, hits.getNumberOfHits());
+ Hit hit = hits.getHit().get(0);
+ assertEquals("the real body", hit.getData(), "a header named contents
must not replace the body");
+ assertEquals(inserted.getExchangeId(),
hit.getDocument().get("exchangeId"),
+ "a header named exchangeId must not replace the exchange id");
+ }
+
+ @Test
+ void sourceFileIsFoundByQueryWithItsPath() throws Exception {
+ // the source directory is indexed when the insert endpoint is created
+ Exchange inserted = template.send("direct:insertWithSrcDir", e ->
e.getIn().setBody("unrelated"));
+ assertNull(inserted.getException(), "insert");
+
+ Hits hits = query("direct:querySrcDir", "sunshine");
+ assertEquals(1, hits.getNumberOfHits(), "a word of a source file must
be found");
+ Document document = hits.getHit().get(0).getDocument();
+ assertNotNull(document, "the hit must carry its document");
+ String path = new File(srcDir, "quote.txt").getPath();
+ assertEquals(path, document.get("path"), "the path must be in the
document of the file contents");
+
+ try (FSDirectory dir = FSDirectory.open(srcIndexDir.toPath());
+ DirectoryReader reader = DirectoryReader.open(dir)) {
+ // the path is not tokenized: the whole path is one term
+ int pathHits = new IndexSearcher(reader).count(new TermQuery(new
Term("path", path)));
+ assertEquals(1, pathHits, "the path must be indexed as one term");
+ }
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index 0c7ffb516af8..36d16341c357 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -3817,6 +3817,25 @@ Applications whose custom code (such as beans calling
`Customer.retrieve(...)` d
`camel-stripe` setting `Stripe.apiKey` globally in the JVM at startup must now
configure `Stripe.apiKey` directly or
pass `RequestOptions` explicitly.
+=== camel-lucene - inserted documents are indexed, one document per exchange
+
+The `lucene:...:insert` endpoint now indexes the fields it writes, so a
`lucene:...:query` finds the inserted messages
+and the files of `srcDir`. Before, the fields were stored but not indexed, and
a query never found them.
+
+*Breaking change:* an index directory written by an earlier Camel version
cannot be appended to. Lucene does not let a
+field change from not indexed to indexed, so the insert fails with
`IllegalArgumentException: cannot change field
+"exchangeId" from index options=NONE to inconsistent index
options=DOCS_AND_FREQS_AND_POSITIONS` (or the same for
+another field). Delete the index directory (`indexDir`) and insert the
messages again. A query on such an index does not
+fail, but it finds none of its documents, as before.
+
+Each exchange is now indexed as one document, with the body in `contents`, the
exchange id in `exchangeId` and each
+header in a field of the header name. Each file of `srcDir` is one document
with the `contents` and `path` fields.
+Before, each of these fields was a document of its own. The files of `srcDir`
are indexed each time the insert endpoint
+is created, such as when the application starts; the document of a file now
replaces the one with the same `path`
+instead of adding a copy. A query on a header, such as `author:carlin`, now
gives a hit
+whose data is the body of the exchange, and with `CamelLuceneReturnLuceneDocs`
the document of a hit carries all the
+fields of its exchange or file.
+
=== camel-wordpress - requests and responses are no longer logged
The WordPress client no longer logs every HTTP request and response it
exchanges with the WordPress server at `INFO`
