[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-24266: - Target Version/s: 3.0.0, 3.1.0, 2.4.7 (was: 2.4.6, 3.0.0, 3.1.0) > Spark client termina

[jira] [Resolved] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

[ https://issues.apache.org/jira/browse/SPARK-31543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31543. -- Resolution: Won't Fix > Backport SPARK-26306 More memory to de-flake SorterSu

[jira] [Resolved] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31538. -- Resolution: Won't Fix > Backport SPARK-25338 Ensure to call super.before

[jira] [Resolved] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

[ https://issues.apache.org/jira/browse/SPARK-31541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31541. -- Resolution: Won't Fix > Backport SPARK-26095 Disable parallelization in m

[jira] [Updated] (SPARK-26908) Fix toMillis

[ https://issues.apache.org/jira/browse/SPARK-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-26908: - Fix Version/s: (was: 2.4.6) 2.4.7 > Fix toMil

[jira] [Commented] (SPARK-26908) Fix toMillis

[ https://issues.apache.org/jira/browse/SPARK-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101926#comment-17101926 ] Holden Karau commented on SPARK-26908: -- retagged to 2.4.7, will revist if we end up cutting an RC2

[jira] [Updated] (SPARK-30737) Reenable to generate Rd files

[ https://issues.apache.org/jira/browse/SPARK-30737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-30737: - Fix Version/s: (was: 2.4.5) 2.4.6 > Reenable to generate Rd fi

[jira] [Updated] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

[ https://issues.apache.org/jira/browse/SPARK-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-27262: - Fix Version/s: (was: 2.4.5) 2.4.6 > Add explicit UTF-8 Encod

[jira] [Assigned] (SPARK-30823) %PYTHONPATH% not set in python/docs/make2.bat, resulting in failed/wrong documentation builds

[ https://issues.apache.org/jira/browse/SPARK-30823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-30823: Assignee: David Toneian > %PYTHONPATH% not set in python/docs/make2.bat, result

[jira] [Updated] (SPARK-30823) %PYTHONPATH% not set in python/docs/make2.bat, resulting in failed/wrong documentation builds

[ https://issues.apache.org/jira/browse/SPARK-30823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-30823: - Fix Version/s: 2.4.6 > %PYTHONPATH% not set in python/docs/make2.bat, resulting in failed/wr

[jira] [Updated] (SPARK-31306) rand() function documentation suggests an inclusive upper bound of 1.0

[ https://issues.apache.org/jira/browse/SPARK-31306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31306: - Fix Version/s: 3.0.0 2.4.6 > rand() function documentation sugge

[jira] [Updated] (SPARK-31231) Support setuptools 46.1.0+ in PySpark packaging

[ https://issues.apache.org/jira/browse/SPARK-31231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31231: - Fix Version/s: 3.1.0 3.0.0 > Support setuptools 46.1.0+ in PySpark packag

[jira] [Updated] (SPARK-31231) Support setuptools 46.1.0+ in PySpark packaging

[ https://issues.apache.org/jira/browse/SPARK-31231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31231: - Fix Version/s: 2.4.6 > Support setuptools 46.1.0+ in PySpark packag

svn commit: r39316 - in /dev/spark/v2.4.6-rc1-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _site/api/java/org/apache/spark

Author: holden Date: Thu May 7 00:48:34 2020 New Revision: 39316 Log: Apache Spark v2.4.6-rc1 docs [This commit notification would consist of 1458 parts, which exceeds the limit of 50 ones, so it was shortened to the summary

svn commit: r39315 - /dev/spark/v2.4.6-rc1-bin/

Author: holden Date: Thu May 7 00:15:41 2020 New Revision: 39315 Log: Apache Spark v2.4.6-rc1 Added: dev/spark/v2.4.6-rc1-bin/ dev/spark/v2.4.6-rc1-bin/SparkR_2.4.6.tar.gz (with props) dev/spark/v2.4.6-rc1-bin/SparkR_2.4.6.tar.gz.asc dev/spark/v2.4.6-rc1-bin/SparkR_2.4.6

[spark] branch branch-2.4 updated (a00eddc -> 8504bdb)

This is an automated email from the ASF dual-hosted git repository. holden pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from a00eddc [SPARK-31653][BUILD] Setuptools is needed before installing any other python packages add a3cffc9

[spark] 01/01: Preparing development version 2.4.7-SNAPSHOT

This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git commit 8504bdb945432479e1a23fccf2308580ba92ecc4 Author: Holden Karau AuthorDate: Wed May 6 23:13:27 2020 +

[spark] 01/01: Preparing Spark release v2.4.6-rc1

This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to tag v2.4.6-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git commit a3cffc997035d11e1f6c092c1186e943f2f63544 Author: Holden Karau AuthorDate: Wed May 6 23:13:21 2020 + Preparing

[spark] tag v2.4.6-rc1 created (now a3cffc9)

This is an automated email from the ASF dual-hosted git repository. holden pushed a change to tag v2.4.6-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git. at a3cffc9 (commit) This tag includes the following new commits: new a3cffc9 Preparing Spark release v2.4.6-rc1

[spark] branch branch-2.4 updated: [SPARK-31653][BUILD] Setuptools is needed before installing any other python packages

This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new a00eddc [SPARK-31653][BUILD] Setuptools

[jira] [Assigned] (SPARK-31653) setuptools needs to be installed before anything else

[ https://issues.apache.org/jira/browse/SPARK-31653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-31653: Assignee: Holden Karau > setuptools needs to be installed before anything e

[jira] [Resolved] (SPARK-31653) setuptools needs to be installed before anything else

[ https://issues.apache.org/jira/browse/SPARK-31653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31653. -- Fix Version/s: 2.4.6 Resolution: Fixed > setuptools needs to be installed bef

[jira] [Updated] (SPARK-31653) setuptools needs to be installed before anything else

[ https://issues.apache.org/jira/browse/SPARK-31653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31653: - Priority: Blocker (was: Major) > setuptools needs to be installed before anything e

[jira] [Created] (SPARK-31653) setuptools needs to be installed before anything else

Holden Karau created SPARK-31653: Summary: setuptools needs to be installed before anything else Key: SPARK-31653 URL: https://issues.apache.org/jira/browse/SPARK-31653 Project: Spark Issue

svn commit: r39313 - /dev/spark/KEYS

Author: holden Date: Wed May 6 20:56:25 2020 New Revision: 39313 Log: Update KEYS Modified: dev/spark/KEYS Modified: dev/spark/KEYS == --- dev/spark/KEYS (original) +++ dev/spark/KEYS Wed May 6 20:56:25 2020

Re: [BVARC] Current Roster

Official as of today: My new vanity call is W5JAZ. Any Donald Fagen fans might recognize that from the song "The Nightfly"! My old call was KI5HTR. Thanks! Bill Holden On Mon, Apr 20, 2020 at 1:27 PM Jeffery MacMillian via BVARC < bvarc@bvarc.org> wrote: > I don't see

Re: Jacek's new Apache Beam Internals Project

over time? > > On Tue, Apr 28, 2020 at 1:48 PM Ismaël Mejía wrote: > >> The tweet URL for ref in case someone wants to like/RT >> >> https://twitter.com/jaceklaskowski/status/1255046717277376512?s=19 >> >> On Tue, Apr 28, 2020, 8:04 PM Holden Karau wrote: >&g

Re: No LICENSE file in spark custom build distribution

Can you send me the output of those two commands On Fri, May 1, 2020 at 1:46 PM Xiangyu Li wrote: > Hi Holden, > > Please check the second email of mine in this email chain. I did that > originally and to qu

Re: No LICENSE file in spark custom build distribution

Your problem isn't the missing license per-se (that just happens to be the first error). I don't believe that is the way we expect users to pip install the Python library. pip will only install directories/targets underneath the directory where setup.py, hence the deps directory which is

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-24266: - Target Version/s: 2.4.6, 3.0.0, 3.1.0 > Spark client terminates while driver is still runn

[jira] [Commented] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096705#comment-17096705 ] Holden Karau commented on SPARK-29048: -- Do we have context for the reason of revert? Should I still

[jira] [Updated] (SPARK-29048) Query optimizer slow when using Column.isInCollection() with a large size collection

[ https://issues.apache.org/jira/browse/SPARK-29048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-29048: - Target Version/s: 2.4.6 Affects Version/s: 2.4.6 2.4.5 > Qu

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096704#comment-17096704 ] Holden Karau commented on SPARK-31538: -- So a test reliability improvement that wouldn't only

Jacek's new Apache Beam Internals Project

Hi Folks, I just saw Jacek's tweet about his new Beam Internals project (he's done a great job on his Spark Internals documentation and blog posts) and I figured I'd share the link https://leanpub.com/the-internals-of-apache-beam in case folks are interested :) Cheers, Holden :) -- Twitter

[jira] [Commented] (SPARK-31525) Inconsistent result of df.head(1) and df.head()

[ https://issues.apache.org/jira/browse/SPARK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094028#comment-17094028 ] Holden Karau commented on SPARK-31525: -- I agree it's inconsistent, also the docs are a little

[jira] [Resolved] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31536. -- Resolution: Won't Fix > Backport SPARK-25407 Allow nested access for non-existent fi

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094024#comment-17094024 ] Holden Karau commented on SPARK-31536: -- If the code path is already disabled by default then yeah

[jira] [Resolved] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31537. -- Resolution: Won't Fix per [~hyukjin.kwon] > Backport SPARK-25559 Remove the unsuppor

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094022#comment-17094022 ] Holden Karau commented on SPARK-31538: -- [~hyukjin.kwon]if the Jira is already closed affecting

[jira] [Commented] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094020#comment-17094020 ] Holden Karau commented on SPARK-31545: -- Sounds reasonable, I'll close this as won't fix

[jira] [Resolved] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31545. -- Resolution: Won't Fix > Backport SPARK-27676 InMemoryFileIndex should resp

Re: Watch "Airbus makes more of the sky with Spark - Jesse Anderson & Hassene Ben Salem" on YouTube

Also it’s ok if Spark and Flink evolve in different directions, were both part of the same open source foundation. Sometimes being everything to everyone isn’t as important as being the best at what you need. I like to think of our relationship with other Apache projects as less competitive and

Re: Copyright Infringment

Glad it makes sense now :) On Sat, Apr 25, 2020 at 11:25 AM Som Lima wrote: > The statement in the book makes sense now. It is based on the premise as > detailed in paragraph 1). > > *unless your are reproducing significant portion of the code. * > > > > > > On

Re: Copyright Infringment

o not want to commit an unlawful act. >>> Can you please clarify if I would be infringing copyright due to this >>> text. >>> *Book: High Performance Spark * >>> *authors: holden Karau Rachel Warren.* >>> *page xii:* >>> >>> * This book is h

[spark] branch master updated: HOTFIX Revert "[SPARK-20732][CORE] Decommission cache blocks

This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9faad07 HOTFIX Revert "[SPARK-20732]

Re: Getting the ball started on a 2.4.6 release

On Fri, Apr 24, 2020 at 6:14 PM edeesis wrote: > Yes, watching the pod yaml could work for this. Just gotta set up some kind > of thing to do that, thanks for clueing me into that. Sure thing, Kris Nova was the one who clued me into it so just passing it along :) > > > And sounds great re:

[jira] [Resolved] (SPARK-20732) Copy cache data when node is being shut down

[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-20732. -- Fix Version/s: 3.1.0 Target Version/s: 3.1.0 Resolution: Fixed Fixed, thank

[jira] [Assigned] (SPARK-20732) Copy cache data when node is being shut down

[ https://issues.apache.org/jira/browse/SPARK-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-20732: Assignee: Prakhar Jain > Copy cache data when node is being shut d

[spark] branch master updated: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

This is an automated email from the ASF dual-hosted git repository. holden pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 249b214 [SPARK-20732][CORE] Decommission cache

[jira] [Created] (SPARK-31555) Improve cache block migration

Holden Karau created SPARK-31555: Summary: Improve cache block migration Key: SPARK-31555 URL: https://issues.apache.org/jira/browse/SPARK-31555 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090954#comment-17090954 ] Holden Karau commented on SPARK-31540: -- I was thinking some folks might build 2.4 with newer JDKs

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090928#comment-17090928 ] Holden Karau commented on SPARK-31544: -- Thanks! > Backport SPARK-30199 Recover `spark.

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090924#comment-17090924 ] Holden Karau commented on SPARK-31540: -- Gotcha, I'll go through the backport JIRAs and link

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090915#comment-17090915 ] Holden Karau commented on SPARK-31539: -- The main reason I'd see to backport the change is it's only

[jira] [Resolved] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31539. -- Resolution: Won't Fix > Backport SPARK-27138 Remove AdminUtils calls (fixes deprecat

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090914#comment-17090914 ] Holden Karau commented on SPARK-31539: -- The other Jira is marked as resolved and I want to track

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Shepherd: Holden Karau > Barrier stage can hang if only partial tasks launc

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Target Version/s: 2.4.6, 3.0.0 > Barrier stage can hang if only partial tasks launc

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090908#comment-17090908 ] Holden Karau commented on SPARK-31536: -- cc [~michael] & [~hyukjin.kwon] for thoughts > B

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090907#comment-17090907 ] Holden Karau commented on SPARK-31538: -- cc [~kiszk] for thoughts > Backport SPARK-25

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090906#comment-17090906 ] Holden Karau commented on SPARK-31539: -- cc [~DylanGuedes] & [~srowen] for thoughts > B

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090905#comment-17090905 ] Holden Karau commented on SPARK-31540: -- cc [~dongjoon] for thoughts > Backport SPARK-27

[jira] [Commented] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

[ https://issues.apache.org/jira/browse/SPARK-31541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090904#comment-17090904 ] Holden Karau commented on SPARK-31541: -- cc [~vanzin] for thoughts > Backport SPARK-26

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090902#comment-17090902 ] Holden Karau commented on SPARK-31542: -- cc [~sanket991] & [~zsxwing] for thoughts > Backpor

[jira] [Commented] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

[ https://issues.apache.org/jira/browse/SPARK-31543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090901#comment-17090901 ] Holden Karau commented on SPARK-31543: -- cc [~gsomogyi] & [~srowen] for thoughts > Backpor

[jira] [Commented] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090900#comment-17090900 ] Holden Karau commented on SPARK-31537: -- cc [~dbtsai]for thoughts > Backport SPARK-25559 Rem

[jira] [Created] (SPARK-31546) Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

Holden Karau created SPARK-31546: Summary: Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled Key: SPARK-31546 URL: https://issues.apache.org/jira/browse/SPARK-31546

[jira] [Updated] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31542: - Component/s: (was: Build) Spark Core > Backport SPARK-25692 Rem

[jira] [Updated] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31545: - Component/s: (was: DStreams) SQL > Backport SPARK-27

[jira] [Created] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

Holden Karau created SPARK-31545: Summary: Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles Key: SPARK-31545 URL: https://issues.apache.org/jira/browse/SPARK-31545

[jira] [Created] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

Holden Karau created SPARK-31544: Summary: Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint Key: SPARK-31544 URL: https://issues.apache.org/jira/browse/SPARK-31544

[jira] [Created] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

Holden Karau created SPARK-31543: Summary: Backport SPARK-26306 More memory to de-flake SorterSuite Key: SPARK-31543 URL: https://issues.apache.org/jira/browse/SPARK-31543 Project: Spark

[jira] [Created] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

Holden Karau created SPARK-31542: Summary: Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well Key

[jira] [Created] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

Holden Karau created SPARK-31541: Summary: Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging) Key: SPARK-31541 URL: https://issues.apache.org/jira/browse/SPARK-31541

[jira] [Created] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

Holden Karau created SPARK-31540: Summary: Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+ Key: SPARK-31540 URL: https://issues.apache.org/jira

[jira] [Created] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

Holden Karau created SPARK-31539: Summary: Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation) Key: SPARK-31539 URL: https://issues.apache.org/jira/browse/SPARK-31539 Project

[jira] [Created] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

Holden Karau created SPARK-31538: Summary: Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases Key: SPARK-31538 URL: https://issues.apache.org/jira/browse/SPARK-31538

[jira] [Created] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

Holden Karau created SPARK-31537: Summary: Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible Key: SPARK-31537 URL: https://issues.apache.org/jira/browse/SPARK-31537

[jira] [Created] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

Holden Karau created SPARK-31536: Summary: Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled Key: SPARK-31536 URL: https://issues.apache.org

Re: Getting the ball started on a 2.4.6 release

up. I do not think it fixes any correctness bugs. > > I think we should discuss your backport plans one by one with the PR > authors and reviewers, since most of them are not closely following the dev > list. > > Xiao > > > On Thu, Apr 23, 2020 at 11:46 AM Holden Kara

Re: Getting the ball started on a 2.4.6 release

I included 26390 as a candidate since it sounded like it bordered on a correctness/expected behaviour fix (eg columpruning rule doing more than column pruning), but if it’s too big a change happy to drop that one. On Thu, Apr 23, 2020 at 11:43 AM Xiao Li wrote: > Hi, Holden, > > We a

Re: Getting the ball started on a 2.4.6 release

listing correctly Not yet merged: SPARK-31485 Barrier execution hang if insufficient resources On Thu, Apr 23, 2020 at 9:13 AM Holden Karau wrote: > > > On Thu, Apr 23, 2020 at 9:07 AM edeesis wrote: > >> There's other information you can obtain from the Pod metadata on a &

Re: Getting the ball started on a 2.4.6 release

On Thu, Apr 23, 2020 at 9:07 AM edeesis wrote: > There's other information you can obtain from the Pod metadata on a > describe > than just from the logs, which are typically what's being printed by the > Application itself. Would get pods -w -o yaml do the trick here or is there going to be

Re: Getting the ball started on a 2.4.6 release

Thanks, I agree improving that error message instead of hanging could be a good candidate for backporting to 2.4 On Tue, Apr 21, 2020 at 6:43 PM wuyi wrote: > I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could > cause application hang. > > > And, probably, also

Re: Getting the ball started on a 2.4.6 release

This seem like not very impactful for end-users on K8s assuming they've got logging of some kind set up. Unless I'm missing something. On Tue, Apr 21, 2020 at 4:51 PM edeesis wrote: > I'd like to advocate for: > > https://issues.apache.org/jira/browse/SPARK-25515 > and >

Re: [pydotorg-www] [Webmaster] Error on "Sunsetting Python 2" Article

Copying to site maintainers. Thanks for your input. Kind regards, Steve On Tue, Apr 21, 2020 at 2:55 PM Corvino, Joseph P < joseph.corv...@duke-energy.com> wrote: > Dear Python Webmaster, > > > > There is an error on the “Sunsetting Python 2” page. In the section titled > “I depend on some

Getting the ball started on a 2.4.6 release

Hi folks, I’m going to get started on putting together a 2.4.6 release to come out hopefully around the same time as 3.0. Are there any changes in master folks think we should be considering backporting to a 2.4.6 release? Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau Books

Re: [BVARC] Current Roster

ian Flores > Bryan Gunderson > William Holden > Karsten Johannessen > Yaroslav Kovalenko Recently EXPIRED Needs to Renew > Daniel Levy > Tina Martinez > Anita Mattis BVARC LIFE MEMBER > David McCay > Christopher McKeon Recently EXPIRED Needs to Renew > Janice Muller > Don

Re: Reference to Beam in upcoming Kubeflow Book

On Fri, Apr 17, 2020 at 3:52 PM Robert Bradshaw wrote: > On Fri, Apr 17, 2020 at 2:56 PM Holden Karau wrote: > >> >> On Fri, Apr 17, 2020 at 2:45 PM Robert Bradshaw >> wrote: >> >>> Hi Holden! >>> >>> I agree with Kyle that it makes se

Re: Reference to Beam in upcoming Kubeflow Book

On Fri, Apr 17, 2020 at 2:45 PM Robert Bradshaw wrote: > Hi Holden! > > I agree with Kyle that it makes sense to have some caveat about Flink and > Spark, though at this point they're not /that/ new (at least not Flink). > True, maybe "early-stage" would be better wordi

Re: Reference to Beam in upcoming Kubeflow Book

On Fri, Apr 17, 2020 at 2:32 PM Ahmet Altay wrote: > Hi Holden, nice to hear from you. Thanks a lot for this email. Adding some > TFX folks as well. +Robert Crowe +Irene > Giannoumis +Zhitao Li +Anusha > Ramesh > > Would it be possible for TFX folks to review the TFX se

Reference to Beam in upcoming Kubeflow Book

open-source runners with language portability. Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Going it alone.

I want to be clear I believe the language in janethrope1s email is unacceptable for the mailing list and possibly a violation of the Apache code of conduct. I’m glad we don’t see messages like this often. I know this is a stressful time for many of us, but let’s try and do our best to not take it

[Python-Dev] Re: PEP 617: New PEG parser for CPython

On Mon, Apr 6, 2020 at 8:04 PM Guido van Rossum wrote: > On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano > wrote: > >> >> Personally, I would not like to have to explain to newcomers why `match` >> is a keyword but you can still use it as a function or variable, but not >> other keywords like

Re: [pydotorg-www] Add me at Python Wiki's Editors Group

HI Leonardo, Thanks very much for wanting to pitch in and help to improve the Wiki! Kind regards, Steve On Sat, Apr 4, 2020 at 10:03 PM Chris Angelico wrote: > On Sun, Apr 5, 2020 at 5:35 AM Leonardo Caballero > wrote: > > > > Hi guys > > > > Please, Add me at Python Wiki's Editors Group >

Re: [pydotorg-www] New editor HansNordhaug

Thanks Hans, You should be good to go! Kind regards, Steve On Fri, Apr 3, 2020 at 1:13 PM Hans Fredrik Nordhaug wrote: > Reply just to you: See mail subject :-) > > Hans > > fre. 3. apr. 2020 kl. 14:09 skrev Steve Holden : > > > > Hi Hans, > > > > Thank

Re: [pydotorg-www] New editor HansNordhaug

Hi Hans, Thanks for your note. If you can let us have your Wiki username we'll be happy to add you to the editors group. Kind regards, Steve On Fri, Apr 3, 2020 at 11:15 AM Hans Fredrik Nordhaug wrote: > (I'm sending this e-mail as recommended on > https://wiki.python.org/moin/FrontPage#use

[jira] [Created] (SPARK-31329) Modify executor monitor to allow for moving shuffle blocks

Holden Karau created SPARK-31329: Summary: Modify executor monitor to allow for moving shuffle blocks Key: SPARK-31329 URL: https://issues.apache.org/jira/browse/SPARK-31329 Project: Spark

Re: Lossless Audio in Games

Re: Lossless Audio in Games I'd like to weigh in here. I'm a little late to the party, but I love flac and would love to see lossless audio in an audiogame. The differences for me are subtle, but I can definitely tell when audio's been compressed. As I say, it's subtle and it's not a giant

[Python-Dev] Re: PEP 616 -- String methods to remove prefixes and suffixes

On Wed, Mar 25, 2020 at 5:42 PM Dennis Sweeney wrote: > I'm removing the tuple feature from this PEP. So now, if I understand > correctly, I don't think there's disagreement about behavior, just about > how that behavior should be summarized in Python code. > [...] > return (the original

<    6   7   8   9   10   11   12   13   14   15   >