Do you have any link or ticket which justifies that k8s does not support spark streaming ?
On Thu, 6 Apr, 2023, 9:15 pm Mich Talebzadeh, <mich.talebza...@gmail.com> wrote: > Do you have a high level diagram of the proposed solution? > > In so far as I know k8s does not support spark structured streaming? > > Mich Talebzadeh, > Lead Solutions Architect/Engineering Lead > Palantir Technologies > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Thu, 6 Apr 2023 at 16:40, Rajesh Katkar <katkar.raj...@gmail.com> > wrote: > >> Use case is , we want to read/write to kinesis streams using k8s >> Officially I could not find the connector or reader for kinesis from >> spark like it has for kafka. >> >> Checking here if anyone used kinesis and spark streaming combination ? >> >> On Thu, 6 Apr, 2023, 7:23 pm Mich Talebzadeh, <mich.talebza...@gmail.com> >> wrote: >> >>> Hi Rajesh, >>> >>> What is the use case for Kinesis here? I have not used it personally, >>> Which use case it concerns >>> >>> https://aws.amazon.com/kinesis/ >>> >>> Can you use something else instead? >>> >>> HTH >>> >>> Mich Talebzadeh, >>> Lead Solutions Architect/Engineering Lead >>> Palantir Technologies >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Thu, 6 Apr 2023 at 13:08, Rajesh Katkar <katkar.raj...@gmail.com> >>> wrote: >>> >>>> Hi Spark Team, >>>> >>>> We need to read/write the kinesis streams using spark streaming. >>>> >>>> We checked the official documentation - >>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>> >>>> It does not mention kinesis connector. Alternative is - >>>> https://github.com/qubole/kinesis-sql which is not active now. This >>>> is now handed over here - https://github.com/roncemer/spark-sql-kinesis >>>> >>>> Also according to SPARK-18165 >>>> <https://issues.apache.org/jira/browse/SPARK-18165> , Spark officially >>>> do not have any kinesis connector >>>> >>>> We have few below questions , It would be great if you can answer >>>> >>>> 1. Does Spark provides officially any kinesis connector which have >>>> readstream/writestream and endorse any connector for production use >>>> cases ? >>>> >>>> 2. >>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>> This >>>> documentation does not mention how to write to kinesis. This method has >>>> default dynamodb as checkpoint, can we override it ? >>>> 3. We have rocksdb as a state store but when we ran an application >>>> using official >>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>> rocksdb >>>> configurations were not effective. Can you please confirm if rocksdb is >>>> not >>>> applicable in these cases? >>>> 4. rocksdb however works with qubole connector , do you have any >>>> plan to release kinesis connector? >>>> 5. Please help/recommend us for any good stable kinesis connector >>>> or some pointers around it >>>> >>>>
