Re: [SS]Kafka EOS transaction timeout solution

Thanks for your reply. 1. About submit tasks to commit kafka transaction kafka producer commit transaction is lightweight action, compared with sending message to kafka, there is little chance to fail in commit transaction. However to implement EOS, we must consider commit transaction fail, and we

[VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

Hi everyone, I'd like to call for a vote on SPARK-28885 "Follow ANSI store assignment rules in table insertion by default". When inserting a value into a column with the different data type, Spark performs type coercion. Currently, we support 3 p

Re: maven 3.6.1 removed from apache maven repo

(Hmm, what is spark-...@apache.org?) From: Sean Owen Sent: Tuesday, September 3, 2019 11:58:30 AM To: Xiao Li Cc: Tom Graves ; spark-...@apache.org Subject: Re: maven 3.6.1 removed from apache maven repo It's because build/mvn only queries ASF mirrors, and the

Re: maven 3.6.1 removed from apache maven repo

It's because build/mvn only queries ASF mirrors, and they remove non-current releases from mirrors regularly (we do the same). This may help avoid this in the future: https://github.com/apache/spark/pull/25667 On Tue, Sep 3, 2019 at 1:41 PM Xiao Li wrote: > Hi, Tom, > > To unblock the build, I m

Re: maven 3.6.1 removed from apache maven repo

Hi, Tom, To unblock the build, I merged the upgrade to master. https://github.com/apache/spark/pull/25665 Thanks! Xiao On Tue, Sep 3, 2019 at 10:58 AM Tom Graves wrote: > It looks like maven 3.6.1 was removed from the repo - see SPARK-28960. It > looks like they pushed 3.6.2, but I don't s

maven 3.6.1 removed from apache maven repo

It looks like maven 3.6.1 was removed from the repo - see SPARK-28960.  It looks like they pushed 3.6.2,  but I don't see any release notes on the maven page for it 3.6.2 Seems like we had this happen before, can't remember if it was maven or something else, anyone remember or know if they are a

Re: Why two netty libs?

Yep, historical reasons. And Netty 4 is under another namespace, so we can use Netty 3 and Netty 4 in the same JVM. On Tue, Sep 3, 2019 at 6:15 AM Sean Owen wrote: > It was for historical reasons; some other transitive dependencies needed > it. > I actually was just able to exclude Netty 3 last

Re: Why two netty libs?

It was for historical reasons; some other transitive dependencies needed it. I actually was just able to exclude Netty 3 last week from master. Spark uses Netty 4. On Tue, Sep 3, 2019 at 6:59 AM Jacek Laskowski wrote: > > Hi, > > Just noticed that Spark 2.4.x uses two netty deps of different vers

Why two netty libs?

Hi, Just noticed that Spark 2.4.x uses two netty deps of different versions. Why? jars/netty-all-4.1.17.Final.jar jars/netty-3.9.9.Final.jar Shouldn't one be excluded or perhaps shaded? Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski The Internals of Spark SQL https://bit.ly/sp

Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for initial offsets?

Hi Devs, Thanks all for a very prompt response! That was insanely quick. Merci beaucoup! :) Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski The Internals of Spark SQL https://bit.ly/spark-sql-internals The Internals of Spark Structured Streaming https://bit.ly/spark-structured-st