GitHub user chiradip edited a discussion: Iggy Connector Architecture Involving
Data/Stream Processors like Flink
Iggy Connector Architecture Involving Data/Stream Processors like Flink
Option X: Flink-Centric Approach
- Develop IggySource/IggySink as a separate Flink connector project (like
flink-connector-kafka)
- This would live in the Apache Flink ecosystem
- Challenge: Requires Flink community acceptance, separate release cycles
Option X.1: Iggy-Housed Flink Connector
- Put the Flink connector inside Iggy's source tree
- New proposal: iggy/core/connectors/external_processors/
- Challenge: How to structure this within Iggy's current architecture
Option Y: Bidirectional Integration (Depends on X)
- Once IggySource/IggySink exist, Flink becomes both a source AND sink for
Iggy
- This creates a bidirectional flow: Iggy → Flink → Iggy
- Enables Flink as a stream processor in Iggy pipelines
Option Z: Iggy-Centric Hub Architecture
- Paradigm shift: Put "Iggy Hub" at the center
- Treat Flink/Spark as pluggable processors, not just sources/sinks
- Current Iggy connectors handle:
- Producers/Consumers (Kafka, databases)
- Queryable systems (APIs, databases)
- Missing piece: Stream processors that transform, multiplex, demultiplex
Current Iggy Connectors handle:
External System → Iggy (Source)
Iggy → External System (Sink)
Missing Category - Stream Processors:
Iggy → Processor (Flink/Spark) → Iggy
(transform/aggregate/join)
Recommended Approach
I propose to start with Option X.1 for these reasons:
1. Immediate control - No external dependencies
2. Faster iteration - Direct integration with Iggy releases
3. Sets foundation for Option Z later
Proposed Structure for X.1:
```
iggy/
├── core/
│ ├── connectors/
│ │ ├── sources/ # Current sources
│ │ ├── sinks/ # Current sinks
│ │ └── processors/ # NEW: Stream processors
│ │ ├── flink/
│ │ │ ├── src/
│ │ │ │ ├── IggyFlinkSource.java
│ │ │ │ ├── IggyFlinkSink.java
│ │ │ │ └── IggyFlinkProcessor.java
│ │ │ ├── Cargo.toml
│ │ │ └── README.md
│ │ └── spark/ # Future
```
This structure:
- Acknowledges processors as a third connector type
- Keeps Flink code within Iggy's control
- Enables future Option Z implementation
- Allows processors to be bidirectional (both source and sink)
The Bigger Vision (Option Z)
Option Z suggests treating Iggy as a stream processing hub, where:
┌──────────────┐
│ Iggy Hub │
└──────┬───────┘
│
┌──────────────────┼──────────────────┐
│ │ │
┌────▼────┐ ┌──────▼──────┐ ┌─────▼────┐
│ Flink │ │ Spark │ │ Custom │
│Processor│ │ Processor │ │Processor │
└─────────┘ └─────────────┘ └──────────┘
This makes Iggy not just a message broker, but a stream processing
orchestrator.
GitHub link: https://github.com/apache/iggy/discussions/2236
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
