On 9/16/19, 8:32 AM, "Chetan Mehrotra" wrote:
Hi Tyson,
> in case of logs NOT in db: when queue full, publish non-blocking to
"completed-non-blocking"
The approach I was thinking was to completely disable (configurable)
support for persisting activation from Invoker
Hi Tyson,
> in case of logs NOT in db: when queue full, publish non-blocking to
> "completed-non-blocking"
The approach I was thinking was to completely disable (configurable)
support for persisting activation from Invoker and instead handle all
such work via activation persister service.
I think this sounds good, but want to be clear I understand the consumers and
producers involved - is this summary correct?
Controller:
* consumes "completed-" topic (as usual)
Invoker:
* in case of logs NOT in db: when queue full, publish non-blocking to
"completed-non-blocking"
*in case of
> For B1, we can scale out the service as controllers are scaled out, but it
> would be much complex to manually assign topics.
Yes thats what my concern was in B1. So would for now target B2
approach where we have a dedicated new topic and then have it consumed
by a new service. If it poses
Let me share a few ideas on them.
Regarding option B1, I think it can scale out better than option B2.
If I understood correctly, scaling out of the service will be highly
dependent on Kafka.
Since the number of consumers is limited to the number of partitions, the
number of service nodes will be
Okie so we can then aim for adding an optional support for storing
activations via a separate service.
Currently we also send the activation result on respective controller
topic. With this change we would also be sending same activation
record on another topic. So we have another choice to make
> Can we handle these in same way as user events? Maybe exactly like user
events, as in use a single service to process both topics.
good call - the user events already contains much of the activation record
(if not all modulo the logs)?
-r
The logs issue is mostly separate from the activation records.
RE activation records:
Can we handle these in same way as user events? Maybe exactly like user events,
as in use a single service to process both topics.
RE logging:
We deal with logs this way (collect from container via fluent),
Rodric Rabbah wrote on 06/20/2019 09:37:38 PM:
>
> Overflowing to Kafka (option b) is better. Actually I would dump all
> the activations there and have a separate process to drain that
> Kafka topic to the datastore or logstore.
I agree. Spilling to Kafka is desirable to avoid OOMs in the
Overflowing to Kafka (option b) is better. Actually I would dump all the
activations there and have a separate process to drain that Kafka topic to the
datastore or logstore.
There is another approach of routing the logs directly to a logstore without
going through the invoker at all. IBM may
Hi Team,
When rate of activation is high (specially with concurrency enabled)
in a specific invoker then its possible that rate of storage of
activation in ArtifactStore lags behind rate of activation record
generation.
For CouchDB this was somewhat mitigated by using a Batcher
implementation
11 matches
Mail list logo
