This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bb59b489204 [SPARK-42205][CORE] Don't log
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c34ec411244 [SPARK-44818] Fix race
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d53ddbe00fe [SPARK-43300][CORE
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5a4b075f95f [SPARK-40261][CORE] Exclude
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 295dd57c13c [SPARK-40235][CORE] Use
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1178bcecc83 [SPARK-40211][CORE][SQL] Allow
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 50c163578cf Revert "[SPARK-4][SQL] Update INSERTs without
user-specified fields to not automaticall
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e17d8ecabca [SPARK-39983][CORE][SQL] Do
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a39fc8773b2 [SPARK-39636][CORE][UI] Fix
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 7fd2e967a8a [SPARK-39422][SQL
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new ff048f1b69e [SPARK-39422][SQL
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8765eea1c08 [SPARK-39422][SQL] Improve error
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 4da8f3a76b1 [SPARK-39361] Don't use
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd45c3656be [SPARK-39361] Don't use Log4J2's
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 02488e0df30a25fd235e2dd25e9b1b3404150125
Author: Tom van Bussel
AuthorDate: Fri Sep 18 11:49:26 2020 +
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit c653d287b15db3e50fa206071a5435028879f15f
Author: sandeepvinayak
AuthorDate: Tue May 31 15:28:07 2022 -0700
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
from 9b268122f68 [SPARK-39293][SQL] Fix the accumulator of ArrayAggregate
to handle complex types properly
new
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 0908337a765 [SPARK-39283][CORE] Fix
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 606830e9cae [SPARK-39283][CORE] Fix
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 1ad1c18fc28 [SPARK-39283][CORE] Fix
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8d0c035f102 [SPARK-39283][CORE] Fix deadlock
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3aaf722 [SPARK-37784][SQL] Correctly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 5cc8b39 [SPARK-37784][SQL] Correctly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 45b7b7e [SPARK-37784][SQL] Correctly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new eeef48fa [SPARK-37784][SQL] Correctly handle
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3b4eb1f [SPARK-37379][SQL] Add tree pattern
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch python-udf-accumulator
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 9213a85a40499fc7f0e24ea14c5051c45a022ef2
Author: Josh Rosen
AuthorDate: Wed Oct 20 16:17:44 2021
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a change to branch python-udf-accumulator
in repository https://gitbox.apache.org/repos/asf/spark.git.
at 9213a85 hacky wip towards python udf profiling
This branch includes the following new commits
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1ef6c13 [SPARK-36933][CORE] Clean up
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1709265 [SPARK-23626][CORE] Eagerly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new c43f355 [SPARK-23626][CORE] Eagerly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 01ee46e [SPARK-23626][CORE] Eagerly
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c4e975e [SPARK-23626][CORE] Eagerly compute
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 3502fda [SPARK-36774][CORE][TESTS
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from f1f2ec3 [SPARK-36735][SQL][FOLLOWUP] Fix indentation of
DynamicPartitionPruningSuite
add 3ae6e67 [SPARK
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 33e45ec [SPARK-28769][CORE] Improve warning message of
BarrierExecutionMode when required slots > maximum sl
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new a71e90a [SPARK-26038][BRANCH-2.4
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ec032ce [SPARK-28112][TEST] Fix Kryo
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ba7f61e [SPARK-26555][SQL][BRANCH
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6b27ad5 [SPARK-28102][CORE] Avoid
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fc65e0f [SPARK-27839][SQL] Change
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 93db7b8 [SPARK-27684][SQL] Avoid conversion
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f90d6dd Update Josh Rosen's
ks.com>
Closes #20191 from
JoshRosen/SPARK-22997-add-defenses-against-use-after-free-bugs-in-memory-allocator.
(cherry picked from commit f340b6b3066033d40b7e163fd5fb68e9820adfb1)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/re
;
Closes #20191 from
JoshRosen/SPARK-22997-add-defenses-against-use-after-free-bugs-in-memory-allocator.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f340b6b3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f340
ork unreliability /
fuzz testing tools.
Author: Josh Rosen <joshro...@databricks.com>
Closes #18662 from JoshRosen/SPARK-21444.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5952ad2b
Tree: http://git-wip-us.apache.org/repos/asf/s
as this patch tested?
Existing tests. I purposely avoided making interface / API which would require
significant updates or modifications to test code.
Author: Josh Rosen <joshro...@databricks.com>
Closes #17955 from JoshRosen/map-output-tracker-rewrite.
Project: http://git-wip-us.apache.org/r
Repository: spark
Updated Branches:
refs/heads/master de953c214 -> 798a04fd7
HOTFIX: fix Scalastyle break introduced in
4d57981cfb18e7500cde6c03ae46c7c9b697d064
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/798a04fd
ipt.
## How was this patch tested?
The LFTP fix was tested by manually running the failing commands on AMPLab
Jenkins against the ASF SFTP server. The PySpark fix was tested locally.
Author: Josh Rosen <joshro...@databricks.com>
Closes #17437 from JoshRosen/spark-20102.
Project: http:
Repository: spark
Updated Branches:
refs/heads/branch-1.6 e78138a43 -> a50ef3d9a
[SPARK-19529][BRANCH-1.6] Backport PR #16866 to branch-1.6
## What changes were proposed in this pull request?
This PR backports PR #16866 to branch-1.6
## How was this patch tested?
Existing tests.
Author:
Repository: spark
Updated Branches:
refs/heads/branch-2.1 80a3e13e5 -> 3b6ac323b
[SPARK-18952][BACKPORT] Regex strings not properly escaped in codegen for
aggregations
## What changes were proposed in this pull request?
Backport for #16361 to 2.1 branch.
## How was this patch tested?
Unit
Repository: spark
Updated Branches:
refs/heads/master 15c2bd01b -> faabe69cc
[SPARK-18952] Regex strings not properly escaped in codegen for aggregations
## What changes were proposed in this pull request?
If I use the function regexp_extract, and then in my regex string, use `\`,
i.e.
sh Rosen <joshro...@databricks.com>
Closes #16189 from JoshRosen/cancellation.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2971ae56
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2971ae56
Diff: http://git-wip-us
viewed #15986.
Author: Josh Rosen <joshro...@databricks.com>
Closes #16070 from JoshRosen/fix-leak-following-total-executor-loss-1.6.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8f25cb26
Tree: http://git-wip-us.apa
hedulerImpl.scala#L523)
in `removeExecutor`, so I'd appreciate a very careful review of these changes.
I added a new unit test to `TaskSchedulerImplSuite`.
/cc kayousterhout and markhamstra, who reviewed #15986.
Author: Josh Rosen <joshro...@databricks.com>
Closes #16045 from JoshRo
or: Josh Rosen <joshro...@databricks.com>
Closes #16045 from JoshRosen/fix-leak-following-total-executor-loss-master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9a02f682
Tree: http://git-wip-us.apache.org/repos/asf/sp
d and merged).
## How was this patch tested?
I added a new unit test to `TaskSchedulerImplSuite`. You can check out this PR
as of 25e455e711b978cd331ee0f484f70fde31307634 to see the failing test.
cc kayousterhout, markhamstra, rxin for review.
Author: Josh Rosen <joshro...@databricks.com>
Closes #1
Repository: spark
Updated Branches:
refs/heads/branch-2.1 951579382 -> 6a3cbbc03
[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed
## What changes were proposed in this pull request?
This PR aims to provide a pip installable PySpark package. This does a bunch of
work to copy the
Repository: spark
Updated Branches:
refs/heads/master bb6cdfd9a -> a36a76ac4
[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed
## What changes were proposed in this pull request?
This PR aims to provide a pip installable PySpark package. This does a bunch of
work to copy the jars
lly tested as part of https://github.com/apache/spark/pull/15659 by having
the build succeed.
cc joshrosen
Author: Holden Karau <hol...@us.ibm.com>
Closes #15860 from holdenk/minor-fix-release-build-script.
(cherry picked from commit 1386fd28daf798bf152606f4da30a36223d75d18)
Signed-off-by: J
ted as part of https://github.com/apache/spark/pull/15659 by having
the build succeed.
cc joshrosen
Author: Holden Karau <hol...@us.ibm.com>
Closes #15860 from holdenk/minor-fix-release-build-script.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.a
ntent.com/assets/50748/19953290/6a271290-a129-11e6-93ad-b825f1448886.png)
Author: Josh Rosen <joshro...@databricks.com>
Closes #15743 from JoshRosen/spark-ui-memory-usage.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3a7
com/typesafehub/migration-manager/issues/115).
Author: Josh Rosen <joshro...@databricks.com>
Closes #15571 from JoshRosen/SPARK-18034.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a65d40ab
Tree: http://git-wip-us.a
com/typesafehub/migration-manager/issues/115).
Author: Josh Rosen <joshro...@databricks.com>
Closes #15571 from JoshRosen/SPARK-18034.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b3b4b954
Tree: http://git-wip-us.apache.
Repository: spark
Updated Branches:
refs/heads/branch-2.0 a2bf09588 -> e355ca8e8
[SPARK-17803][TESTS] Upgrade docker-client dependency
[SPARK-17803: Docker integration tests don't run with "Docker for
Mac"](https://issues.apache.org/jira/browse/SPARK-17803)
## What changes were proposed in
Repository: spark
Updated Branches:
refs/heads/master 9a48e60e6 -> 49d11d499
[SPARK-17803][TESTS] Upgrade docker-client dependency
[SPARK-17803: Docker integration tests don't run with "Docker for
Mac"](https://issues.apache.org/jira/browse/SPARK-17803)
## What changes were proposed in this
nce any columns.
## How was this patch tested?
New regression test in FilterPushdownSuite.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15289 from JoshRosen/SPARK-17712.
(cherry picked from commit 37eb9184f1e9f1c07142c66936671f4711ef407d)
Signed-off-by: Josh Rosen <joshro...@da
Repository: spark
Updated Branches:
refs/heads/branch-2.0 ca8130050 -> 7ffafa3bf
[SPARK-16343][SQL] Improve the PushDownPredicate rule to pushdown predicates
correctly in non-deterministic condition.
## What changes were proposed in this pull request?
Currently our Optimizer may reorder the
Repository: spark
Updated Branches:
refs/heads/branch-2.0 98bbc4410 -> 2cd327ef5
[SPARK-17056][CORE] Fix a wrong assert regarding unroll memory in MemoryStore
## What changes were proposed in this pull request?
There is an assert in MemoryStore's putIteratorAsValues method which is used to
Repository: spark
Updated Branches:
refs/heads/master 2f84a6866 -> e7bce9e18
[SPARK-17056][CORE] Fix a wrong assert regarding unroll memory in MemoryStore
## What changes were proposed in this pull request?
There is an assert in MemoryStore's putIteratorAsValues method which is used to
row an IllegalArgumentException if it is called with
an object that is not an `UnsafeRow`.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15265 from JoshRosen/SPARK-17618-master.
(cherry picked from commit 2f84a686604b298537bfd4d087b41594d2aa7ec6)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
ion if it is called with
an object that is not an `UnsafeRow`.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15265 from JoshRosen/SPARK-17618-master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2f84a686
T
Repository: spark
Updated Branches:
refs/heads/branch-2.0 f14f47f07 -> 243bdb11d
[SPARK-17613] S3A base paths with no '/' at the end return empty DataFrames
Consider you have a bucket as `s3a://some-bucket`
and under it you have files:
```
s3a://some-bucket/file1.parquet
Repository: spark
Updated Branches:
refs/heads/master 9f24a17c5 -> 85d609cf2
[SPARK-17613] S3A base paths with no '/' at the end return empty DataFrames
## What changes were proposed in this pull request?
Consider you have a bucket as `s3a://some-bucket`
and under it you have files:
```
5186 from JoshRosen/SPARK-17485-branch-1.6-backport.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/94524cef
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/94524cef
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff
joshro...@databricks.com>
Closes #15167 from JoshRosen/stop-publishing-kinesis-assembly.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ce0a222f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ce0a222f
Diff: http:
joshro...@databricks.com>
Closes #15167 from JoshRosen/stop-publishing-kinesis-assembly.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d7ee1221
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d7ee1221
Diff: http://git-wip-us.a
5156 from JoshRosen/SPARK-17160.
(cherry picked from commit e719b1c045ba185d242d21bbfcdee2c84dafc587)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7026eb87
Tree:
ror message string
literals in generated Java code, leading to compilation errors.
This patch addresses these issues by using `addReferenceObj` to store the error
messages as string fields rather than inline string constants.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15156 from
Repository: spark
Updated Branches:
refs/heads/branch-2.0 c4660d607 -> f56035ba6
[SPARK-17473][SQL] fixing docker integration tests error due to different
versions of jars.
## What changes were proposed in this pull request?
Docker tests are using older version of jersey jars (1.19), which
Repository: spark
Updated Branches:
refs/heads/master d720a4019 -> cdea1d134
[SPARK-17473][SQL] fixing docker integration tests error due to different
versions of jars.
## What changes were proposed in this pull request?
Docker tests are using older version of jersey jars (1.19), which was
ith `zip` but
didn't first check that the lengths of the two collections were equal, causing
missing records to go unnoticed. The updated test case reproduced this bug.
In addition, I added a new `PartiallySerializedBlockSuite` to unit test that
component.
Author: Josh Rosen <joshro...@databricks.com>
ith `zip` but
didn't first check that the lengths of the two collections were equal, causing
missing records to go unnoticed. The updated test case reproduced this bug.
In addition, I added a new `PartiallySerializedBlockSuite` to unit test that
component.
Author: Josh Rosen <joshro...@databrick
ted?
Two new regression tests in BlockManagerSuite.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15085 from JoshRosen/SPARK-17484.
(cherry picked from commit 1202075c95eabba0ffebc170077df798f271a139)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
Project:
ted?
Two new regression tests in BlockManagerSuite.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15085 from JoshRosen/SPARK-17484.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1202075c
Tree: http://git-wip-us.a
ion of the temp file.
This patch avoids this potential cause of disk-space leaks by adding `finally`
blocks to ensure that temp files are always deleted if they haven't been
renamed.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuff
ion of the temp file.
This patch avoids this potential cause of disk-space leaks by adding `finally`
blocks to ensure that temp files are always deleted if they haven't been
renamed.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuff
ion of the temp file.
This patch avoids this potential cause of disk-space leaks by adding `finally`
blocks to ensure that temp files are always deleted if they haven't been
renamed.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15104 from JoshRosen/cleanup-tmp-data-file-in-shuffle-writer.
Repository: spark
Updated Branches:
refs/heads/branch-2.0 fffcec90b -> bb2bdb440
[SPARK-17465][SPARK CORE] Inappropriate memory management in
`org.apache.spark.storage.MemoryStore` may lead to memory leak
The expression like `if (memoryMap(taskAttemptId) == 0)
Repository: spark
Updated Branches:
refs/heads/master dbfc7aa4d -> bb3229436
[SPARK-17465][SPARK CORE] Inappropriate memory management in
`org.apache.spark.storage.MemoryStore` may lead to memory leak
The expression like `if (memoryMap(taskAttemptId) == 0)
memoryMap.remove(taskAttemptId)`
Repository: spark
Updated Branches:
refs/heads/branch-1.6 bf3f6d2f1 -> a447cd888
[SPARK-17465][SPARK CORE] Inappropriate memory management in
`org.apache.spark.storage.MemoryStore` may lead to memory leak
## What changes were proposed in this pull request?
The expression like `if
Repository: spark
Updated Branches:
refs/heads/master ff6e4cbdc -> e33bfaed3
[SPARK-17463][CORE] Make CollectionAccumulator and SetAccumulator's value can
be read thread-safely
## What changes were proposed in this pull request?
Make CollectionAccumulator and SetAccumulator's value can be
Repository: spark
Updated Branches:
refs/heads/branch-2.0 fab77dadf -> fffcec90b
[SPARK-17463][CORE] Make CollectionAccumulator and SetAccumulator's value can
be read thread-safely
## What changes were proposed in this pull request?
Make CollectionAccumulator and SetAccumulator's value can
tes` returned `None` when no remote
locations for the block could be found (which could occur if an executor died
and its block manager de-registered with the master).
Author: Josh Rosen <joshro...@databricks.com>
Closes #15037 from JoshRosen/SPARK-17485.
(cherry picked fr
sen <joshro...@databricks.com>
Closes #15061 from JoshRosen/post-2.0-mima-changes.
(cherry picked from commit 7c51b99a428a965ff7d136e1cdda20305d260453)
Signed-off-by: Josh Rosen <joshro...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wi
joshro...@databricks.com>
Closes #15061 from JoshRosen/post-2.0-mima-changes.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7c51b99a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7c51b99a
Diff: http:
ere into their own
separate PR in order to make them easier to review and so that the
behavior-changing parts of my other patch can be isolated to their own PR.
Author: Josh Rosen <joshro...@databricks.com>
Closes #15036 from JoshRosen/cache-failure-race-conditions-refactorings-only.
Repository: spark
Updated Branches:
refs/heads/branch-2.0 30521522d -> 0a36e360c
[SPARK-17503][CORE] Fix memory leak in Memory store when unable to cache the
whole RDD in memory
## What changes were proposed in this pull request?
MemoryStore may throw OutOfMemoryError when trying to
Repository: spark
Updated Branches:
refs/heads/master 8087ecf8d -> 1742c3ab8
[SPARK-17503][CORE] Fix memory leak in Memory store when unable to cache the
whole RDD in memory
## What changes were proposed in this pull request?
MemoryStore may throw OutOfMemoryError when trying to cache a
ask for the
first-level hash map storage, even when running in low-memory situations such
as local mode. This changes it to use the memory manager default page size,
which is automatically reduced from 64MB in these situations.
cc ooq JoshRosen
## How was this patch tested?
Tested manually with `bin/sp
1 - 100 of 928 matches
Mail list logo
