Tim Allison created TIKA-5001:
---------------------------------
Summary: Improve runtime configuration in grpc
Key: TIKA-5001
URL: https://issues.apache.org/jira/browse/TIKA-5001
Project: Tika
Issue Type: Task
Reporter: Tim Allison
>From my bot, based on:
{noformat}
Summary. Runtime fetcher/emitter registration (gRPC SaveFetcher/DeleteFetcher)
is only partly wired through tika-pipes: config-file components are copied into
the
config store with resolved secrets, updates and deletes never reach a fork
that has already built the instance, and no fetcher or emitter is ever closed.
1. Keep config-file components out of the store (staged in v2)
- Before: every process writes every config-file fetcher/emitter into the
store at startup, ${env:...} resolved, so a file store holds secrets in
plaintext and a shared
Ignite store holds every node's config. A runtime save over a
config-defined id is undone the next time a fork starts.
- After: config-file components live in memory only; the store holds only
runtime registrations as the caller sent them. A config-defined id can't be
replaced or
deleted at runtime; register a new id. Legacy store copies are ignored with
a WARN. Upgrade note: delete store entries you didn't register at runtime. 2.
Managers own component lifecycle (lease and close)
- Before: one instance per id cached forever; nothing ever closes a fetcher
or emitter, not on save/delete, not on shutdown (JDBCEmitter.close() never
called).
- After: acquire(id) returns a lease held for the whole task and released in
its finally. A replaced/deleted instance is retired at once and closed when its
last lease
is released (immediately if idle). Everything is closed on graceful
shutdown. Closeable components get called. API change: getFetcher/getEmitter
return a lease. Docs:
shared mode requires thread-safe fetchers and emitters. 3. Propagate store
changes to running forks
- Before: with file/ignite a fork picks up a new id on a cache miss, then
caches it until restart, so updates and deletes are invisible. With memory,
SaveFetcher
succeeds and ListFetchers shows it, but every FetchAndParse fails
FETCHER_NOT_FOUND.
- After: per task, the fork compares the store fingerprint (file: content
hash; ignite: counter) with the last seen; on change it retires all runtime
components via
step 2. Updates and deletes apply to the next task; a running task finishes
with what it holds. Startup fails on allowComponentManagement: true +
configStoreType:
memory. 4. Shipped emitter bugs (own issues, linked)
- KafkaEmitter: async send(), never flushed or acked; buffered records die
with a recycled or killed fork. After: wait for the ack per emit; producer
closed via step 2.
- JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
shared mode can interleave parameters across rows. After: serialize the emit or
use a
connection per thread. {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
