GitHub user chiradip edited a discussion: Apache Iggy Connector for Apache Pinot
# Apache Iggy Connector for Apache Pinot This connector enables Apache Pinot to ingest real-time data from Apache Iggy streams using TCP-based communication. ## Overview The Iggy Pinot connector implements Pinot's Stream Plugin API to provide: - **TCP-based ingestion**: Uses Iggy's native TCP protocol via `AsyncIggyTcpClient` for efficient message consumption - **Partition-aware consumption**: Supports parallel ingestion from multiple Iggy partitions - **Consumer group support**: Leverages Iggy consumer groups for offset management and fault tolerance - **Automatic offset management**: Consumer group state is maintained by Iggy server - **JSON message decoding**: Built-in support for JSON-formatted messages ## Architecture ### Key Components 1. **IggyConsumerFactory**: Main entry point implementing Pinot's `StreamConsumerFactory` 2. **IggyPartitionGroupConsumer**: Partition-level consumer using TCP client 3. **IggyStreamMetadataProvider**: Provides partition discovery and offset information 4. **IggyJsonMessageDecoder**: Decodes JSON messages into Pinot records ### Differences from Flink Connector The Pinot connector differs from the Iggy Flink connector in several ways: - **TCP-based**: Uses `AsyncIggyTcpClient` directly (not HTTP-based) - **Consumer group managed**: Relies on Iggy's consumer group offset management - **Simpler offset handling**: No custom offset storage, leverages Iggy's built-in state - **Pinot-specific APIs**: Implements `PartitionGroupConsumer` instead of Flink's `SourceReader` GitHub link: https://github.com/apache/iggy/discussions/2449 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
