Re: [VOTE] Release Spark 2.4.6 (RC1)

I confirmed and update the JIRA. SPARK-31663 is a correctness issue since Apache Spark 2.4.0. Bests, Dongjoon. On Fri, May 8, 2020 at 10:26 AM Holden Karau wrote: > Can you provide a bit more context (is it a regression?) > > On Fri, May 8, 2020 at 9:33 AM Yuanjian Li wrote: > >> Hi Holden,

Re: [VOTE] Release Spark 2.4.6 (RC1)

Can you provide a bit more context (is it a regression?) On Fri, May 8, 2020 at 9:33 AM Yuanjian Li wrote: > Hi Holden, > > I'm working on the bugfix of SPARK-31663 > , let me post it here > since it's a correctness bug and also affects 2.4.6.

Re: [VOTE] Release Spark 2.4.6 (RC1)

Hi Holden, I'm working on the bugfix of SPARK-31663 , let me post it here since it's a correctness bug and also affects 2.4.6. Best, Yuanjian Sean Owen 于2020年5月8日周五 下午11:42写道： > +1 from me. The usual: sigs OK, license looks as intended, tests

Re: Inconsistent schema on Encoders.bean (reported issues from user@)

The first case of user report is obvious - according to the user report, AVRO generated code contains getter which denotes to itself hence Spark disallows (throws exception), but it doesn't have matching setter method (if I understand correctly) so technically it shouldn't matter. For the second

Re: [VOTE] Release Spark 2.4.6 (RC1)

+1 from me. The usual: sigs OK, license looks as intended, tests pass from a source build for me. On Thu, May 7, 2020 at 1:29 PM Holden Karau wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.6. > > The vote is open until February 5th 11PM PST and passes

Re: [DISCUSS] Resolve ambiguous parser rule between two "create table"s

Hi all, I'd like to bring this up again to share the status and get more feedback. Currently, we all agree to unify the CREATE TABLE syntax by merging the native and Hive-style syntaxes. The unified CREATE TABLE syntax will become the native syntax and there is no Hive-style syntax anymore. This

Re: Inconsistent schema on Encoders.bean (reported issues from user@)

Can you give some simple examples to demonstrate the problem? I think the inconsistency would bring problems but don't know how. On Fri, May 8, 2020 at 3:49 PM Jungtaek Lim wrote: > (bump to expose the discussion to more readers) > > On Mon, May 4, 2020 at 4:57 PM Jungtaek Lim > wrote: > >> Hi

Re: Handling user-facing metadata issues on file stream source & sink

(bump to expose the discussion to more readers) On Mon, May 4, 2020 at 5:45 PM Jungtaek Lim wrote: > Hi devs, > > I'm seeing more and more structured streaming end users encountered the > metadata issues on file stream source and sink. They have been known-issues > and there're even

Re: Inconsistent schema on Encoders.bean (reported issues from user@)

(bump to expose the discussion to more readers) On Mon, May 4, 2020 at 4:57 PM Jungtaek Lim wrote: > Hi devs, > > There're couple of issues being reported on the user@ mailing list which > results in being affected by inconsistent schema on Encoders.bean. > > 1. Typed datataset from Avro

Executor exceptions stacktrace omitted by HotSpot in long running application

Hi, I'm considering to improve the experience of hitting potential exceptions stacktrace omitted in long running application[1], which is a JVM HotSpot optimization as Shixiong(Ryan) commented[2]. There might be 2 options: 1. Adds `-XX:-OmitStackTraceInFastThrow` as a common Executor JVM option.