rohankumardubey opened a new issue, #4127:
URL: https://github.com/apache/iggy/issues/4127
### Description
### Description
Add an Apache Spark Structured Streaming source integration so Spark
applications can consume Apache Iggy streams through the DataFrame and
Dataset APIs.
This integration is listed in #2753 under Stream Processing Integration,
but it does not currently have a dedicated issue or implementation.
The initial scope would include:
- A Spark Structured Streaming source, rather than the legacy DStream
receiver API.
- A Java or Scala connector module, following the existing Flink
external-processor structure where reusable.
- Mapping Iggy stream partitions to Spark input partitions.
- Representing Iggy offsets through Spark's streaming offset API.
- Committing Iggy progress only after Spark commits the completed
micro-batch.
- Exposing the message payload and relevant Iggy metadata, including
stream, topic, partition, offset, and timestamp.
- Configuration for server addresses, authentication, TLS, stream,
topic, consumer identity or group, starting offset, and batch limits.
- Unit tests, documentation, and an integration example using local
Spark and Iggy.
The initial implementation would not include a Spark sink, legacy
DStream support, or continuous-processing mode. Those can be addressed
separately after the source semantics are validated.
### Affected area/component
Connectors and Java SDK.
### Proposed solution
Implement a Spark Data Source V2 micro-batch source, potentially under:
`foreign/java/external-processors/iggy-connector-spark`
The connector would implement Spark's `MicroBatchStream` integration
surface. Iggy partitions would be represented as Spark input
partitions, and Iggy offsets would be represented as Spark streaming
offsets. Spark checkpointing and the micro-batch commit callback would
be used to coordinate recovery and offset advancement.
The existing Flink connector can provide reusable project structure,
configuration, Java SDK, serialization, testing, and example patterns.
The delivery and recovery guarantees should be explicitly documented
and tested, particularly around retries after a Spark executor,
application, or Iggy connection failure.
### Alternatives considered
- Using Iggy's Kafka compatibility layer. This would be less native and
would couple the integration to Kafka-compatible behavior.
- Implementing a legacy Spark Streaming DStream receiver. Structured
Streaming is Spark's current integration model and provides the
appropriate offset and checkpoint interfaces.
- Using a raw TCP or socket source. This would not provide the necessary
partition and offset semantics.
### Contribution
I am willing to implement this after the scope and design are confirmed.
### Affected area / component
_No response_
### Proposed solution
_No response_
### Alternatives considered
_No response_
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
