2019-10-30 09:18:49 UTC - James Meen: @James Meen has joined the channel ---- 2019-10-30 11:10:47 UTC - Kabeer Ahmed: @Kabeer Ahmed has joined the channel ---- 2019-10-30 11:21:23 UTC - Kabeer Ahmed: Hello!
I am looking at: <https://github.com/streamnative/pulsar-spark/commit/9bfdbe90279e3436912be01c4210363f278d29d5#diff-600376dffeb79835ede4a0b285078036>. This is the first commit from history of the pom.xml file. The spark version that is used is 2.4.0. Obviously with Spark 2.3, the build fails with the error below: ``` [ERROR] [Error] /home/kabeer/code/apache/pulsar-spark/src/main/scala/org/apache/spark/sql/pulsar/PulsarSource.scala:25: object JSONOptionsInRead is not a member of package org.apache.spark.sql.catalyst.json ``` Is there anyone with some knowledge about this? Just for the sake of pulsar, we do not want to yet migrate to Spark 2.4 and would be great to still use Spark 2.3.4? Thanks, ---- 2019-10-30 11:31:04 UTC - Sijie Guo: @Kabeer Ahmed thank you for reporting this. it seems that there are class changes between Spark 2.3.4 and 2.4.0. It should be quite simple to fix it. @yijie can help with that. ---- 2019-10-30 11:48:57 UTC - Kabeer Ahmed: @Sijie Guo - Thank you for quick response. Yes you are right. There have been new classes added in Spark 2.4. If you look into file for Spark 2.3 -> <https://github.com/apache/spark/blob/branch-2.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala>, class JSONOptionsInRead is missing. Whilst in Spark 2.4 -> <https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala>, class JSONOptionsInRead is present. Any help with this is highly appreciated! If I get a patch, I am more than happy to test it out. Do you advise raising a jira against the project? Let me know if you want me to. ---- 2019-10-30 11:50:16 UTC - Sijie Guo: Creating a github issue against that project woulld be great! ---- 2019-10-30 11:50:43 UTC - Sijie Guo: I think @yijie starts looking into it. ---- 2019-10-30 11:51:08 UTC - Kabeer Ahmed: @Sijie Guo I shall raise the ticket now. ---- 2019-10-30 11:51:56 UTC - yijie: ok. please feel free to file it ---- 2019-10-30 12:00:11 UTC - Kabeer Ahmed: @yijie - Issue opened now: <https://github.com/streamnative/pulsar-spark/issues/23> +1 : yijie, Sijie Guo ---- 2019-10-30 12:00:53 UTC - yijie: OK ---- 2019-10-30 12:57:54 UTC - Ryan: @Ryan has joined the channel ---- 2019-10-30 13:35:58 UTC - lookers: @lookers has joined the channel ---- 2019-10-31 00:03:01 UTC - Sireesha: @Sireesha has joined the channel ---- 2019-10-31 00:21:26 UTC - Ryan: In light of this discussion, what is the relationship or plans for Kafka + Pulsar? ---- 2019-10-31 08:47:10 UTC - Jasper Li: @Jasper Li has joined the channel ---- 2019-10-31 08:49:32 UTC - Jacky Kwok: @Jacky Kwok has joined the channel ---- 2019-10-31 08:55:04 UTC - Jasper Li: Hello Everyone! I want to ask which files I should modify if I want to configure tired storage of Pulsar cluster running on GKE with S3? I can only find an article of the standalone pulsar: <https://streaml.io/blog/configuring-apache-pulsar-tiered-storage-with-amazon-s3>. Thanks a lot! ---- 2019-10-31 09:02:01 UTC - xiaolong.ran: <https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/> ----
