[
https://issues.apache.org/jira/browse/TIKA-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Allison updated TIKA-5001:
------------------------------
Description:
>From my bot, based on:
{noformat}
Summary. Runtime fetcher/emitter registration (gRPC SaveFetcher/DeleteFetcher)
is only partly wired
through tika-pipes: config-file components are copied into the config store
with resolved
secrets, updates and deletes never reach a fork that has already built the
instance,
and no fetcher or emitter is ever closed.
Steps
1. Keep config-file components out of the store
- Before: every process writes every config-file fetcher/emitter into the
store at startup,
${env:...} resolved, so a file store holds secrets in plaintext and a
shared Ignite store
holds every node's config. A runtime save over a config-defined id is
undone the next
time a fork starts.
- After: config-file components live in memory only; the store holds only
runtime registrations
as the caller sent them. A config-defined id can't be replaced or deleted
at runtime;
register a new id. Legacy store copies are ignored with a WARN. Upgrade
note: delete store
entries you didn't register at runtime.
2. Managers own component lifecycle (lease and close)
- Before: one instance per id cached forever; nothing ever closes a fetcher
or emitter, not on
save/delete, not on shutdown (JDBCEmitter.close() never called).
- After: acquire(id) returns a lease held for the whole task and released in
its finally. A
replaced/deleted instance is retired at once and closed when its last lease
is released
(immediately if idle). Everything is closed on graceful shutdown. Closeable
components get
called. API change: getFetcher/getEmitter return a lease. Docs: shared mode
requires
thread-safe fetchers and emitters.
3. Propagate store changes to running forks
- Before: with file/ignite a fork picks up a new id on a cache miss, then
caches it until
restart, so updates and deletes are invisible. With memory, SaveFetcher
succeeds and ListFetchers shows it, but every FetchAndParse fails
FETCHER_NOT_FOUND.
- After: per task, the fork compares the store fingerprint (file: content
hash; ignite: counter)
with the last seen; on change it retires all runtime components via step 2.
Updates
and deletes apply to the next task; a running task finishes with what it
holds.
Startup fails on allowComponentManagement: true + configStoreType: memory.
4. Shipped emitter bugs (own issues, linked)
- KafkaEmitter: async send(), never flushed or acked; buffered records die
with a recycled
or killed fork. After: wait for the ack per emit; producer closed via step
2.
- JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
shared mode can
interleave parameters across rows. After: serialize the emit or use a
connection per thread. {noformat}
was:
>From my bot, based on:
{noformat}
Summary. Runtime fetcher/emitter registration (gRPC SaveFetcher/DeleteFetcher)
is only partly wired through tika-pipes: config-file components are copied into
the
config store with resolved secrets, updates and deletes never reach a fork
that has already built the instance, and no fetcher or emitter is ever closed.
1. Keep config-file components out of the store (staged in v2)
- Before: every process writes every config-file fetcher/emitter into the
store at startup, ${env:...} resolved, so a file store holds secrets in
plaintext and a shared
Ignite store holds every node's config. A runtime save over a
config-defined id is undone the next time a fork starts.
- After: config-file components live in memory only; the store holds only
runtime registrations as the caller sent them. A config-defined id can't be
replaced or
deleted at runtime; register a new id. Legacy store copies are ignored with
a WARN. Upgrade note: delete store entries you didn't register at runtime. 2.
Managers own component lifecycle (lease and close)
- Before: one instance per id cached forever; nothing ever closes a fetcher
or emitter, not on save/delete, not on shutdown (JDBCEmitter.close() never
called).
- After: acquire(id) returns a lease held for the whole task and released in
its finally. A replaced/deleted instance is retired at once and closed when its
last lease
is released (immediately if idle). Everything is closed on graceful
shutdown. Closeable components get called. API change: getFetcher/getEmitter
return a lease. Docs:
shared mode requires thread-safe fetchers and emitters. 3. Propagate store
changes to running forks
- Before: with file/ignite a fork picks up a new id on a cache miss, then
caches it until restart, so updates and deletes are invisible. With memory,
SaveFetcher
succeeds and ListFetchers shows it, but every FetchAndParse fails
FETCHER_NOT_FOUND.
- After: per task, the fork compares the store fingerprint (file: content
hash; ignite: counter) with the last seen; on change it retires all runtime
components via
step 2. Updates and deletes apply to the next task; a running task finishes
with what it holds. Startup fails on allowComponentManagement: true +
configStoreType:
memory. 4. Shipped emitter bugs (own issues, linked)
- KafkaEmitter: async send(), never flushed or acked; buffered records die
with a recycled or killed fork. After: wait for the ack per emit; producer
closed via step 2.
- JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
shared mode can interleave parameters across rows. After: serialize the emit or
use a
connection per thread. {noformat}
> Improve runtime configuration in grpc
> -------------------------------------
>
> Key: TIKA-5001
> URL: https://issues.apache.org/jira/browse/TIKA-5001
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> From my bot, based on:
>
> {noformat}
> Summary. Runtime fetcher/emitter registration (gRPC
> SaveFetcher/DeleteFetcher) is only partly wired
> through tika-pipes: config-file components are copied into the config store
> with resolved
> secrets, updates and deletes never reach a fork that has already built the
> instance,
> and no fetcher or emitter is ever closed.
> Steps
> 1. Keep config-file components out of the store
> - Before: every process writes every config-file fetcher/emitter into the
> store at startup,
> ${env:...} resolved, so a file store holds secrets in plaintext and a
> shared Ignite store
> holds every node's config. A runtime save over a config-defined id is
> undone the next
> time a fork starts.
> - After: config-file components live in memory only; the store holds only
> runtime registrations
> as the caller sent them. A config-defined id can't be replaced or deleted
> at runtime;
> register a new id. Legacy store copies are ignored with a WARN. Upgrade
> note: delete store
> entries you didn't register at runtime.
> 2. Managers own component lifecycle (lease and close)
> - Before: one instance per id cached forever; nothing ever closes a fetcher
> or emitter, not on
> save/delete, not on shutdown (JDBCEmitter.close() never called).
> - After: acquire(id) returns a lease held for the whole task and released
> in its finally. A
> replaced/deleted instance is retired at once and closed when its last
> lease is released
> (immediately if idle). Everything is closed on graceful shutdown.
> Closeable components get
> called. API change: getFetcher/getEmitter return a lease. Docs: shared
> mode requires
> thread-safe fetchers and emitters.
> 3. Propagate store changes to running forks
> - Before: with file/ignite a fork picks up a new id on a cache miss, then
> caches it until
> restart, so updates and deletes are invisible. With memory, SaveFetcher
> succeeds and ListFetchers shows it, but every FetchAndParse fails
> FETCHER_NOT_FOUND.
> - After: per task, the fork compares the store fingerprint (file: content
> hash; ignite: counter)
> with the last seen; on change it retires all runtime components via step
> 2. Updates
> and deletes apply to the next task; a running task finishes with what it
> holds.
> Startup fails on allowComponentManagement: true + configStoreType:
> memory.
> 4. Shipped emitter bugs (own issues, linked)
> - KafkaEmitter: async send(), never flushed or acked; buffered records die
> with a recycled
> or killed fork. After: wait for the ack per emit; producer closed via
> step 2.
> - JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
> shared mode can
> interleave parameters across rows. After: serialize the emit or use a
> connection per thread. {noformat}
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
