GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/7943
[SPARK-9439] [wip] [yarn] External shuffle service robust to NM restarts
using leveldb
an alternative to https://github.com/apache/spark/pull/7839, storing the
data in leveldb, rather than just in a regular file.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark
leveldb_external_shuffle_service_NM_restart
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7943.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7943
----
commit b9d2ced92a9712edf7612f4cee1ac929e5fa3dab
Author: Imran Rashid <[email protected]>
Date: 2015-07-28T14:17:15Z
incomplete setup for external shuffle service tests
commit 36127d38750e75d682c8858b7128bd634732be71
Author: Imran Rashid <[email protected]>
Date: 2015-07-28T18:27:02Z
wip
commit bb3ba494be05a11ab8105ce4ce570f60d1d33e51
Author: Imran Rashid <[email protected]>
Date: 2015-07-28T18:27:12Z
minor cleanup
commit c69f46b98b6b9da16bfc7fc3a15deeade3e229ed
Author: Imran Rashid <[email protected]>
Date: 2015-07-30T20:08:49Z
maybe working version, needs tests & cleanup ...
commit 5e5a7c36c3f6ac2c0f0dd6738eb4bb66605f8428
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T15:37:14Z
fix build
commit ad122ef6b07eba0326929057704ef827a7eb26d6
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T15:40:00Z
more fixes
commit 0b588bd5e0dd4ee81f9b4f856f9a57f4a4c72151
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T15:52:57Z
more fixes ...
commit 1136f44eb9bca70c365db8e004d7b84840f9dc55
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T20:33:56Z
test needs to have an actual shuffle
commit 9eae11939a408639ec990b36407b1a330d20961a
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T20:43:44Z
cleanup lots of duplication
commit 0e9d69b9fc4bdebd5d226778c336f3157f3b065f
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T21:04:48Z
better names
commit d596969df600adcc291f56ab9c105ffcf3218253
Author: Imran Rashid <[email protected]>
Date: 2015-07-31T21:28:00Z
cleanup imports
commit dd93dc033429cdbeae72a02bffb0d45ac7401dee
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T04:37:57Z
test for shuffle service w/ NM restarts
commit efb61958fd037952f987cfb08062c331946800b6
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T05:13:04Z
proper unit test, and no longer leak if apps stop during NM restart
commit a36729c7b7dbb9120bc7c57d08c48766c30ddc65
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T05:13:44Z
cleanup
commit 7504de8ffde710d4f81f7c53a15d11c2874c1a99
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T07:01:21Z
style
commit 23994ff0fd84162b150b2cb379eea0f8c1d54e5d
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T07:02:01Z
style
commit 86e0cb9aeebaa3ba25cb35ac1f5e78541b6a2af2
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T14:15:13Z
for tests, shuffle service finds an open port
commit bdc4b32125a53a0f7a9e5b714a49d44af35b3a56
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T14:15:36Z
rename
commit bb9d1e6f02a1c40c6df435c7380c161bd929116e
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T14:17:07Z
formatting
commit 857331a6a755db705e6c24a6d29acf4fe5c2fcc5
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T19:49:55Z
better tests & comments
commit bdbbf0dd190007c776753869189164fb0b24bca3
Author: Imran Rashid <[email protected]>
Date: 2015-08-01T20:09:43Z
comments, remove some unnecessary changes
commit 62586a68e3ec3e22aad58d7b74599371eb352b30
Author: Imran Rashid <[email protected]>
Date: 2015-08-03T19:16:47Z
just serialize the whole executors map
commit 245db19e5ac43dc6717da508f015498e07e09cad
Author: Imran Rashid <[email protected]>
Date: 2015-08-03T19:17:25Z
style
commit 55f49fc9865100cc45cdeb37c578ddb9ce5849bf
Author: Imran Rashid <[email protected]>
Date: 2015-08-03T21:54:02Z
make sure the service doesnt die if the registered executor file is
corrupt; add tests
commit 0a39b98599afacc2a9b5ec70f5e7d68a10b601e5
Author: Imran Rashid <[email protected]>
Date: 2015-08-03T22:03:25Z
Merge branch 'master' into external_shuffle_service_NM_restart
Conflicts:
core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala
commit 44928353fd630b46cdc3e97c76f3a7593955539a
Author: Imran Rashid <[email protected]>
Date: 2015-08-03T22:17:21Z
lol, dont use a PrintWriter b/c of scalastyle checks
commit f729e2b6aac60f380a337e1bf0b42858006ab7b0
Author: Imran Rashid <[email protected]>
Date: 2015-08-04T00:53:01Z
debugging
commit d7450f0e3209c776d333090836ffc57de8c3b34c
Author: Imran Rashid <[email protected]>
Date: 2015-08-04T01:10:27Z
style
commit 32fe5ae2f41cb8d27cea0d09b9b87efd1d4d1d26
Author: Imran Rashid <[email protected]>
Date: 2015-08-04T03:04:43Z
Merge branch 'master' into external_shuffle_service_NM_restart
commit 59800b77f0597608691af1323880f54c6a269ef4
Author: Imran Rashid <[email protected]>
Date: 2015-08-04T20:23:25Z
Files.move in case renaming is unsupported
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
