[GitHub] jzhuge commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

jzhuge commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165#issuecomment-452937118 Sure. On Wed, Jan 9, 2019 at 5:18 PM Sean Owen wrote: > Oops, good catch @jzhuge

Re: Remove non-Tungsten mode in Spark 3?

I haven't touched Tungsten, but have proposed removing the deprecated old memory manager and settings -- yes I think that's the primary argument for it. https://github.com/apache/spark/pull/23457 On Wed, Jan 9, 2019 at 6:06 PM Erik Erlandson wrote: > Removing the user facing config seems like a

[GitHub] srowen commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

srowen commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165#issuecomment-452934048 Oops, good catch @jzhuge . Can you run `jekyll build` locally to also update the HTML? if it's any trouble I can do it

Re: [VOTE] SPARK 2.2.3 (RC1)

Hm OK those other profiles should be unrelated. I'll see if I can figure it out, but it's likely this is specific to the machine I am testing on somehow. For that reason, I'll say +1 on the basis that these tests really do pass. On Wed, Jan 9, 2019 at 6:05 PM Dongjoon Hyun wrote: > > I tested

Re: DataSourceV2 community sync tonight

There are 2 remaining problems in the write side API refactor : 1. how to put the `queryId` parameter in the write API 2. how to put the streaming OutputMode parameter in the write API I'd like to

[GitHub] jzhuge opened a new pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

jzhuge opened a new pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165 This is an automated message from the Apache Git Service. To respond

Re: SPARk-25299: Updates As Of December 19, 2018

Curious how SPARK-25299 (where file tracking is pushed to spark drivers, at least in option-5) interacts with Splash. The shuffle data location in SPARK-25299 would now have additional "fallback" logic for recovering from executor loss. On Thu, Jan 3, 2019 at 6:24 AM Peter Rudenko wrote: > Hi

Re: Remove non-Tungsten mode in Spark 3?

Removing the user facing config seems like a good idea from the standpoint of reducing cognitive load, and documentation On Fri, Jan 4, 2019 at 7:03 AM Sean Owen wrote: > OK, maybe leave in tungsten for 3.0. > I did a quick check, and removing StaticMemoryManager saves a few hundred > lines.

Re: [VOTE] SPARK 2.2.3 (RC1)

I tested with Maven and `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver` on CentOS/JDK8. The difference seems to be `-Pmesos -Psparkr` from your and `-Pkinesis-asl` from mine. Do you think it's related? BTW, at least, we have a green balls on Jenkins.

Re: [VOTE] SPARK 2.2.3 (RC1)

BTW did you run with the same profiles, I wonder; I test with, generally, -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Psparkr I am checking mostly because none of that weird error would happen without testing hive-thriftserver. The others are probably just flakiness or something else

Re: [VOTE] SPARK 2.2.3 (RC1)

Hi, Sean. It looks strange. I didn't hit them. I'm not sure but it looks like some flakiness at 2.2.x era. For me, those test passes. (I ran twice before starting a vote and during this voting from the source tar file) Bests, Dongjoon On Wed, Jan 9, 2019 at 1:42 PM Sean Owen wrote: > I wonder

Re: [VOTE] SPARK 2.2.3 (RC1)

I wonder if anyone else is seeing the following issues, or whether it's specific to my environment: With -Phive-thriftserver, it compiles fine. However during tests, I get ... [error]

DataSourceV2 community sync tonight

Hi everyone, This is a quick reminder that there is a DSv2 community sync tonight at 5 PM PST. These community syncs are open to anyone that wants to participate. If you’d like to be added to the invite, please send me a direct message. The main topic for this sync is the catalog API. To make

Re: [VOTE] SPARK 2.2.3 (RC1)

+1 On Wed, Jan 9, 2019 at 4:30 AM Dongjoon Hyun wrote: > +1 > > Bests, > Dongjoon. > > On Tue, Jan 8, 2019 at 6:30 PM Wenchen Fan wrote: > >> +1 >> >> On Wed, Jan 9, 2019 at 3:37 AM DB Tsai wrote: >> >>> +1 >>> >>> Sincerely, >>> >>> DB Tsai >>>

Re: [VOTE] SPARK 2.2.3 (RC1)

+1 Bests, Dongjoon. On Tue, Jan 8, 2019 at 6:30 PM Wenchen Fan wrote: > +1 > > On Wed, Jan 9, 2019 at 3:37 AM DB Tsai wrote: > >> +1 >> >> Sincerely, >> >> DB Tsai >> -- >> Web: https://www.dbtsai.com >> PGP Key ID: 0x5CED8B896A6BDFA0 >>

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Jörn, may you explain a bit more your proposal, please? We are not modifying the existing decimal datatype. This is how it works now. If you check the PR, the only difference is how we compute the result for the divsion operation. The discussion about precision and scale is about: shall we limit

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Maybe it is better to introduce a new datatype that supports negative scale, otherwise the migration and testing efforts for organizations running Spark application becomes too large. Of course the current decimal will be kept as it is. > Am 07.01.2019 um 15:08 schrieb Marco Gaido : > > In

Re: [DISCUSS] Support decimals with negative scale in decimal operation

Oracle does the same: "The *scale* must be less than or equal to the precision." (see https://docs.oracle.com/javadb/10.6.2.1/ref/rrefsqlj15260.html). Il giorno mer 9 gen 2019 alle ore 05:31 Wenchen Fan ha scritto: > Some more thoughts. If we support unlimited negative scale, why can't we >