[
https://issues.apache.org/jira/browse/TIKA-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132625#comment-18132625
]
ASF GitHub Bot commented on TIKA-5001:
--------------------------------------
tballison opened a new pull request, #3436:
URL: https://github.com/apache/tika/pull/3436
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for your contribution to [Apache Tika](https://tika.apache.org/)!
Your help is appreciated!
Before opening the pull request, please verify that
* there is an open issue on the [Tika issue
tracker](https://issues.apache.org/jira/projects/TIKA) which describes the
problem or the improvement. We cannot accept pull requests without an issue
because the change wouldn't be listed in the release notes.
* the issue ID (`TIKA-XXXX`)
- is referenced in the title of the pull request
- and placed in front of your commit messages surrounded by square
brackets (`[TIKA-XXXX] Issue or pull request title`)
* commits are squashed into a single one (or few commits for larger changes)
* Tika builds and unit tests pass with `./mvnw clean install` (`clean test`
alone cannot resolve the pipes plugin zips)
* if you used a generative AI tool: follow the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) (`Generated-by:
<tool>` in the commit message), and consider running the pre-flight in
`.skills/devs/pr-review/SKILL.md` — fix what it finds; don't paste its report
here
* there should be no conflicts when merging the pull request branch into the
*recent* `main` branch. If there are conflicts, please try to rebase the pull
request branch on top of a freshly pulled `main` branch
* if you add new module that downstream users will depend upon add it to
relevant group in `tika-bom/pom.xml`.
We will be able to faster integrate your pull request if these conditions
are met. If you have any questions how to fix your problem or about using Tika
in general, please sign up for the [Tika mailing
list](http://tika.apache.org/mail-lists.html). Thanks!
> Improve runtime configuration in grpc
> -------------------------------------
>
> Key: TIKA-5001
> URL: https://issues.apache.org/jira/browse/TIKA-5001
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> From my bot, based on ({*}[https://github.com/apache/tika/pull/3411]){*}
>
> {noformat}
> Summary. Runtime fetcher/emitter registration (gRPC
> SaveFetcher/DeleteFetcher) is only partly wired
> through tika-pipes: config-file components are copied into the config store
> with resolved
> secrets, updates and deletes never reach a fork that has already built the
> instance,
> and no fetcher or emitter is ever closed.
> Steps
> 1. Keep config-file components out of the store
> - Before: every process writes every config-file fetcher/emitter into the
> store at startup,
> ${env:...} resolved, so a file store holds secrets in plaintext and a
> shared Ignite store
> holds every node's config. A runtime save over a config-defined id is
> undone the next
> time a fork starts.
> - After: config-file components live in memory only; the store holds only
> runtime registrations
> as the caller sent them. A config-defined id can't be replaced or deleted
> at runtime;
> register a new id. Legacy store copies are ignored with a WARN. Upgrade
> note: delete store
> entries you didn't register at runtime.
> 2. Managers own component lifecycle (lease and close)
> - Before: one instance per id cached forever; nothing ever closes a fetcher
> or emitter, not on
> save/delete, not on shutdown (JDBCEmitter.close() never called).
> - After: acquire(id) returns a lease held for the whole task and released
> in its finally. A
> replaced/deleted instance is retired at once and closed when its last
> lease is released
> (immediately if idle). Everything is closed on graceful shutdown.
> Closeable components get
> called. API change: getFetcher/getEmitter return a lease. Docs: shared
> mode requires
> thread-safe fetchers and emitters.
> 3. Propagate store changes to running forks
> - Before: with file/ignite a fork picks up a new id on a cache miss, then
> caches it until
> restart, so updates and deletes are invisible. With memory, SaveFetcher
> succeeds and ListFetchers shows it, but every FetchAndParse fails
> FETCHER_NOT_FOUND.
> - After: per task, the fork compares the store fingerprint (file: content
> hash; ignite: counter)
> with the last seen; on change it retires all runtime components via step
> 2. Updates
> and deletes apply to the next task; a running task finishes with what it
> holds.
> Startup fails on allowComponentManagement: true + configStoreType:
> memory.
> 4. Shipped emitter bugs (own issues, linked)
> - KafkaEmitter: async send(), never flushed or acked; buffered records die
> with a recycled
> or killed fork. After: wait for the ack per emit; producer closed via
> step 2.
> - JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
> shared mode can
> interleave parameters across rows. After: serialize the emit or use a
> connection per thread. {noformat}
>
> ||Store||Runtime registrations live in||Fork sees a new id||Fork sees an
> update or delete||Holds config-file copies (resolved secrets)||Survives gRPC
> restart||Broken today||
> |{{memory}} (default)|the gRPC JVM's heap only|No: {{SaveFetcher}} succeeds,
> {{ListFetchers}} shows it, every {{FetchAndParse}} through it fails
> {{FETCHER_NOT_FOUND}}|No|n/a (per process)|No|Yes, with
> {{allowComponentManagement}}; fine without it. Fixed by step 3 (startup fails
> on the combination)|
> |{{file}}|a JSON file every process reads and writes|Yes, on the fork's first
> cache miss|No, until the fork restarts|Yes, plaintext on disk|Yes|Yes:
> secrets on disk (step 1), stale updates/deletes (steps 2, 3), nothing closed
> (step 2)|
> |{{ignite}} (opt-in jars)|an Ignite cache shared across processes and
> nodes|Yes, on the fork's first cache miss|No, until the fork restarts|Yes,
> every node's config in
> one cache|Only with Ignite persistence|Yes: stale updates/deletes (steps 2,
> 3), nothing closed (step 2), cross-node config copies (step 1)|
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
