unsubscribe On Wed, Apr 12, 2023 at 3:59 PM Rajesh Katkar <katkar.raj...@gmail.com> wrote:
> Hi Lingzhe, > > We are also started using this operator. > Do you see any issues with it? > > > On Wed, 12 Apr, 2023, 7:25 am Lingzhe Sun, <lingzhe....@hirain.com> wrote: > >> Hi Mich, >> >> FYI we're using spark operator( >> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) to build >> stateful structured streaming on k8s for a year. Haven't test it using >> non-operator way. >> >> Besides that, the main contributor of the spark operator, Yinan Li, has >> been inactive for quite long time. Kind of worried that this project might >> finally become outdated as k8s is evolving. So if anyone is interested, >> please support the project. >> >> ------------------------------ >> Lingzhe Sun >> Hirain Technologies >> >> >> *From:* Mich Talebzadeh <mich.talebza...@gmail.com> >> *Date:* 2023-04-11 02:06 >> *To:* Rajesh Katkar <katkar.raj...@gmail.com> >> *CC:* user <u...@spark.incubator.apache.org> >> *Subject:* Re: spark streaming and kinesis integration >> What I said was this >> "In so far as I know k8s does not support spark structured streaming?" >> >> So it is an open question. I just recalled it. I have not tested myself. >> I know structured streaming works on Google Dataproc cluster but I have not >> seen any official link that says Spark Structured Streaming is supported on >> k8s. >> >> HTH >> >> Mich Talebzadeh, >> Lead Solutions Architect/Engineering Lead >> Palantir Technologies >> London >> United Kingdom >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Mon, 10 Apr 2023 at 06:31, Rajesh Katkar <katkar.raj...@gmail.com> >> wrote: >> >>> Do you have any link or ticket which justifies that k8s does not support >>> spark streaming ? >>> >>> On Thu, 6 Apr, 2023, 9:15 pm Mich Talebzadeh, <mich.talebza...@gmail.com> >>> wrote: >>> >>>> Do you have a high level diagram of the proposed solution? >>>> >>>> In so far as I know k8s does not support spark structured streaming? >>>> >>>> Mich Talebzadeh, >>>> Lead Solutions Architect/Engineering Lead >>>> Palantir Technologies >>>> London >>>> United Kingdom >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Thu, 6 Apr 2023 at 16:40, Rajesh Katkar <katkar.raj...@gmail.com> >>>> wrote: >>>> >>>>> Use case is , we want to read/write to kinesis streams using k8s >>>>> Officially I could not find the connector or reader for kinesis from >>>>> spark like it has for kafka. >>>>> >>>>> Checking here if anyone used kinesis and spark streaming combination ? >>>>> >>>>> On Thu, 6 Apr, 2023, 7:23 pm Mich Talebzadeh, < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Hi Rajesh, >>>>>> >>>>>> What is the use case for Kinesis here? I have not used it personally, >>>>>> Which use case it concerns >>>>>> >>>>>> https://aws.amazon.com/kinesis/ >>>>>> >>>>>> Can you use something else instead? >>>>>> >>>>>> HTH >>>>>> >>>>>> Mich Talebzadeh, >>>>>> Lead Solutions Architect/Engineering Lead >>>>>> Palantir Technologies >>>>>> London >>>>>> United Kingdom >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, 6 Apr 2023 at 13:08, Rajesh Katkar <katkar.raj...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Spark Team, >>>>>>> >>>>>>> We need to read/write the kinesis streams using spark streaming. >>>>>>> >>>>>>> We checked the official documentation - >>>>>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>>>>> >>>>>>> It does not mention kinesis connector. Alternative is - >>>>>>> https://github.com/qubole/kinesis-sql which is not active now. >>>>>>> This is now handed over here - >>>>>>> https://github.com/roncemer/spark-sql-kinesis >>>>>>> >>>>>>> Also according to SPARK-18165 >>>>>>> <https://issues.apache.org/jira/browse/SPARK-18165> , Spark >>>>>>> officially do not have any kinesis connector >>>>>>> >>>>>>> We have few below questions , It would be great if you can answer >>>>>>> >>>>>>> 1. Does Spark provides officially any kinesis connector which >>>>>>> have readstream/writestream and endorse any connector for production >>>>>>> use >>>>>>> cases ? >>>>>>> 2. >>>>>>> >>>>>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>>>>> This >>>>>>> documentation does not mention how to write to kinesis. This method >>>>>>> has >>>>>>> default dynamodb as checkpoint, can we override it ? >>>>>>> 3. We have rocksdb as a state store but when we ran an >>>>>>> application using official >>>>>>> >>>>>>> https://spark.apache.org/docs/latest/streaming-kinesis-integration.html >>>>>>> rocksdb >>>>>>> configurations were not effective. Can you please confirm if rocksdb >>>>>>> is not >>>>>>> applicable in these cases? >>>>>>> 4. rocksdb however works with qubole connector , do you have any >>>>>>> plan to release kinesis connector? >>>>>>> 5. Please help/recommend us for any good stable kinesis >>>>>>> connector or some pointers around it >>>>>>> >>>>>>>
