[
https://issues.apache.org/jira/browse/TIKA-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132596#comment-18132596
]
Tim Allison commented on TIKA-5001:
-----------------------------------
If I had my agent do steps 1-4 as above, would that be useful, or is this the
wrong direction? [~kristian]
> Improve runtime configuration in grpc
> -------------------------------------
>
> Key: TIKA-5001
> URL: https://issues.apache.org/jira/browse/TIKA-5001
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> From my bot, based on ({*}[https://github.com/apache/tika/pull/3411]){*}
>
> {noformat}
> Summary. Runtime fetcher/emitter registration (gRPC
> SaveFetcher/DeleteFetcher) is only partly wired
> through tika-pipes: config-file components are copied into the config store
> with resolved
> secrets, updates and deletes never reach a fork that has already built the
> instance,
> and no fetcher or emitter is ever closed.
> Steps
> 1. Keep config-file components out of the store
> - Before: every process writes every config-file fetcher/emitter into the
> store at startup,
> ${env:...} resolved, so a file store holds secrets in plaintext and a
> shared Ignite store
> holds every node's config. A runtime save over a config-defined id is
> undone the next
> time a fork starts.
> - After: config-file components live in memory only; the store holds only
> runtime registrations
> as the caller sent them. A config-defined id can't be replaced or deleted
> at runtime;
> register a new id. Legacy store copies are ignored with a WARN. Upgrade
> note: delete store
> entries you didn't register at runtime.
> 2. Managers own component lifecycle (lease and close)
> - Before: one instance per id cached forever; nothing ever closes a fetcher
> or emitter, not on
> save/delete, not on shutdown (JDBCEmitter.close() never called).
> - After: acquire(id) returns a lease held for the whole task and released
> in its finally. A
> replaced/deleted instance is retired at once and closed when its last
> lease is released
> (immediately if idle). Everything is closed on graceful shutdown.
> Closeable components get
> called. API change: getFetcher/getEmitter return a lease. Docs: shared
> mode requires
> thread-safe fetchers and emitters.
> 3. Propagate store changes to running forks
> - Before: with file/ignite a fork picks up a new id on a cache miss, then
> caches it until
> restart, so updates and deletes are invisible. With memory, SaveFetcher
> succeeds and ListFetchers shows it, but every FetchAndParse fails
> FETCHER_NOT_FOUND.
> - After: per task, the fork compares the store fingerprint (file: content
> hash; ignite: counter)
> with the last seen; on change it retires all runtime components via step
> 2. Updates
> and deletes apply to the next task; a running task finishes with what it
> holds.
> Startup fails on allowComponentManagement: true + configStoreType:
> memory.
> 4. Shipped emitter bugs (own issues, linked)
> - KafkaEmitter: async send(), never flushed or acked; buffered records die
> with a recycled
> or killed fork. After: wait for the ack per emit; producer closed via
> step 2.
> - JDBCEmitter: concurrent emits share one PreparedStatement with no lock;
> shared mode can
> interleave parameters across rows. After: serialize the emit or use a
> connection per thread. {noformat}
>
> ||Store||Runtime registrations live in||Fork sees a new id||Fork sees an
> update or delete||Holds config-file copies (resolved secrets)||Survives gRPC
> restart||Broken today||
> |{{memory}} (default)|the gRPC JVM's heap only|No: {{SaveFetcher}} succeeds,
> {{ListFetchers}} shows it, every {{FetchAndParse}} through it fails
> {{FETCHER_NOT_FOUND}}|No|n/a (per process)|No|Yes, with
> {{allowComponentManagement}}; fine without it. Fixed by step 3 (startup fails
> on the combination)|
> |{{file}}|a JSON file every process reads and writes|Yes, on the fork's first
> cache miss|No, until the fork restarts|Yes, plaintext on disk|Yes|Yes:
> secrets on disk (step 1), stale updates/deletes (steps 2, 3), nothing closed
> (step 2)|
> |{{ignite}} (opt-in jars)|an Ignite cache shared across processes and
> nodes|Yes, on the fork's first cache miss|No, until the fork restarts|Yes,
> every node's config in
> one cache|Only with Ignite persistence|Yes: stale updates/deletes (steps 2,
> 3), nothing closed (step 2), cross-node config copies (step 1)|
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
