Copilot commented on code in PR #3174:
URL: https://github.com/apache/tika/pull/3174#discussion_r4018816598
##########
tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIImageEmbeddingParser.java:
##########
@@ -185,9 +191,10 @@ public void parse(TikaInputStream tis, ContentHandler
handler,
@Override
public void initialize() throws TikaConfigException {
- LOG.info("openai-image-embedding-parser runs one request per image;
the \"engines\" + "
- + "\"inference\" shape (openai-embedding-engine, input IMAGES,
task embed) batches "
- + "a document's images into one request");
+ LOG.warn("openai-image-embedding-parser is deprecated since 4.1.0 and
will be removed in "
+ + "4.2.0: configure the endpoint as an openai-embedding-engine
under \"engines\" "
+ + "and bind it with an IMAGES \"inference\" binding (task
embed), which sends a "
+ + "document's images in requests of the engine's batch size");
Review Comment:
`ParserLoader.finish()` now emits `warnEngineUnderParsers` for every
`ContentEnricher` under `parsers`, but this `initialize()` method also emits a
deprecation WARN for the same parser. Configuring
`openai-image-embedding-parser` under `parsers` therefore produces two startup
WARNs, despite the documented one-WARN-per-entry behavior; coordinate these
paths so direct `text-recognizers` use remains covered without duplicating the
legacy-shape warning.
##########
tika-serialization/src/main/java/org/apache/tika/config/loader/ParserLoader.java:
##########
@@ -205,66 +205,71 @@ private static String names(List<Parser> parsers) {
return names.toString();
}
- /**
- * Enrichers named directly under {@code "parsers"} that the composite
never dispatches
- * to: every type they advertise is claimed by another parser there, or
they advertise
- * none (engine unavailable, or told to skip). Both shapes look configured
and do
- * nothing as parsers.
- */
- static List<Parser> undispatchedEnrichers(Parser root) {
- List<Parser> inert = new ArrayList<>();
+ /** Engines (enrichers) named directly under {@code "parsers"}: the
deprecated 4.0 shape. */
+ static List<Parser> enginesUnderParsers(Parser root) {
+ List<Parser> engines = new ArrayList<>();
if (!(root instanceof CompositeParser composite) || root instanceof
DefaultParser) {
- return inert;
+ return engines;
+ }
+ for (Parser member : composite.getAllComponentParsers()) {
+ if (ContentEnrichers.isEnricher(member)) {
+ engines.add(member);
+ }
+ }
+ return engines;
+ }
+
+ /** The types the composite dispatches to this member as their parser. */
+ static Set<MediaType> parsedTypes(Parser root, Parser member) {
+ Set<MediaType> parsed = new TreeSet<>();
+ if (!(root instanceof CompositeParser composite)) {
+ return parsed;
}
ParseContext empty = new ParseContext();
Map<MediaType, Parser> dispatch = composite.getParsers(empty);
MediaTypeRegistry registry = composite.getMediaTypeRegistry();
- for (Parser member : composite.getAllComponentParsers()) {
- if (!ContentEnrichers.isEnricher(member)) {
- continue;
- }
- boolean dispatched = false;
- for (MediaType type : member.getSupportedTypes(empty)) {
- if (dispatch.get(registry.normalize(type)) == member) {
- dispatched = true;
- break;
- }
- }
- if (!dispatched) {
- inert.add(member);
+ for (MediaType type : member.getSupportedTypes(empty)) {
+ if (dispatch.get(registry.normalize(type)) == member) {
+ parsed.add(type);
}
}
- return inert;
+ return parsed;
}
- // the 4.0 shape still works, so it is INFO; an entry that never runs at
all is a WARN
- private static void logUndispatched(Parser inert, CompositeContentEnricher
enrichers,
- boolean listConfigured) {
- String name = ParserUtils.getParserClassname(inert);
- Set<MediaType> advertised = inert.getSupportedTypes(new
ParseContext());
+ /** One WARN per engine under "parsers": the deprecation, then what the
entry does today. */
+ private static void warnEngineUnderParsers(Parser engine, Parser root,
+ CompositeContentEnricher
enrichers,
+ boolean listConfigured) {
+ String name = ParserUtils.getParserClassname(engine);
+ String lead = name + " is named under \"parsers\", which is deprecated
for engines since "
+ + "4.1.0 and unsupported in 4.2.0: configure it under
\"engines\" and name it in "
+ + "\"text-recognizers\". ";
Review Comment:
This guidance is not correct for every `ContentEnricher` found under
`parsers`: `OpenAIImageEmbeddingParser` is an annotator, not a
`TextRecognizer`, and its replacement is an `openai-embedding-engine` with an
`IMAGES` `inference` binding. The current WARN tells users to put it in
`text-recognizers` (and the later branches call it a text recognizer), so make
the recommendation depend on the engine/binding or state both supported
migration paths.
##########
docs/modules/ROOT/pages/configuration/inference.adoc:
##########
@@ -231,9 +231,10 @@ request" (see
xref:configuration/inference-recipes.adoc[Recipes]). Names only, s
xref:using-tika/server/index.adoc[preset]. A name not in the configured list
fails the request.
Engines cannot be defined or changed per request.
-== From `openai-image-embedding-parser`
+== From `openai-image-embedding-parser` (deprecated)
-The 4.0 `openai-image-embedding-parser` still works as a text-recognizers
entry and embeds one
-image per request. The same endpoint as an engine plus a binding on `IMAGES`
embeds a
-document's images in one request and puts the vectors where they belong; move
the `baseUrl`,
-`model` and `apiKey` to the engine and drop the parser entry.
+The 4.0 `openai-image-embedding-parser` is deprecated since 4.1.0 and removed
in 4.2.0. It
+still works as a text-recognizers entry, with a WARN at startup, and embeds
one image per
+request. The same endpoint as an engine plus a binding on `IMAGES` embeds a
document's images
+in one request and puts the vectors where they belong; move the `baseUrl`,
`model` and
+`apiKey` to the engine and drop the parser entry.
Review Comment:
An `IMAGES` binding batches units up to the engine's `maxBatchSize`, so a
document with more images than that produces multiple requests. Saying the
replacement embeds all images "in one request" is inaccurate and conflicts with
the engine's batching behavior; describe batched requests instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
