Hello Fineract developers,

I have recently started exploring and contributing to Apache Fineract
Business Intelligence, particularly the Kafka CDC ingestion and
control-plane components.

While studying the connector reconciliation workflow, I noticed a potential
lifecycle-safety issue that I would like to validate with the maintainers
before attempting any changes.

*Observation: Connector generation replacement*

In controlplane/connectors.py, the plan_actions() function processes
desired connectors first and then schedules deletion of managed connectors
that are no longer desired.

I noticed a few scenarios worth investigating:

   1.

   When a replacement connector is missing, its name is added to
   desired_by_cluster, but if max_creates_per_run is reached, its creation
   is deferred.
   2.

   The previous-generation connector is not part of the desired connector
   set and may still be scheduled for deletion.
   3.

   During execution in controlplane/reconciler.py, an exception from a
   connector action is recorded, but execution continues with subsequent
   actions. Therefore, a failed replacement creation may not prevent the old
   connector's deletion.
   4.

   The execution path for CREATE submits the connector configuration, but
   the code inspected does not appear to wait for the replacement task to
   reach a healthy running state before deleting the previous connector.

This raises a question about the intended replacement invariant:

Should the previous connector remain available until its replacement has
been successfully created and reached an acceptable operational state?

I'd be interested in contributing regression tests for these scenarios and,
depending on the intended behavior, a fix to the reconciliation workflow.

*Related architectural observation*

While tracing DLT handling, I also noticed that connector names and
replication slots are generation-specific, whereas the final Kafka topics
use a shared table-oriented namespace. The current ChangeEvent and DLT
metadata we inspected do not appear to preserve the original capture
generation.

This may have implications for recovering events after a source connector
is recreated. I think this is a separate design question and would be happy
to discuss it independently.

I haven't made any implementation changes yet, as I'd first like to
understand the maintainers' expectations and whether these behaviors are
already known or covered by another design decision.

Would this be a useful area for contribution? I'm happy to investigate
further, add regression tests, and work on an agreed solution.

Thank you!
Yours Sincerely
Samarth

Reply via email to