GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/8320
[SPARK-6236] [core] [wip] caching for blocks over 2GB
This allows caching of partitions that are greater than 2GB, and also
includes error messages for the other cases where we hit the 2GB limit
(replication & shuffle).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-1391_2g_partition_limit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8320.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8320
----
commit 5cdcd4246e586346a8e1ac2242dd795fdb1ae068
Author: Imran Rashid <[email protected]>
Date: 2015-02-20T22:35:12Z
add some failing tests, though these probably shouldnt actually get merged
commit 03db862833f3c4feef2d72620bc5c9a893dab2f5
Author: Imran Rashid <[email protected]>
Date: 2015-02-23T20:28:22Z
steal some code from earlier work of @mridulm
commit d6337f03a4ac2971a004ef821281723e857f9008
Author: Imran Rashid <[email protected]>
Date: 2015-02-24T18:18:38Z
wip -- changed a bunch of types to LargeByteBuffer; discovered problem on
replicate()
commit a139e97fe1aeac279b9c47119745c0f45eb7d8c5
Author: Imran Rashid <[email protected]>
Date: 2015-02-25T19:27:13Z
compiling but all sorts of bad casting etc.
commit 4965bad00574a05e133e7caeba56cd6115fe35b6
Author: Imran Rashid <[email protected]>
Date: 2015-02-25T20:28:16Z
move LargeByteBuffer to network-common, since we need it there for the
shuffles
commit 149d4fa3fa55403df90109b440a3523d3f4ab92b
Author: Imran Rashid <[email protected]>
Date: 2015-02-25T21:50:33Z
move large byte buffer to network/common ... still lots of crud
commit 01cafbf15026fdcbfd58566335802082493a491c
Author: Imran Rashid <[email protected]>
Date: 2015-02-25T21:53:22Z
tests compile too
commit ce391a0dbbba3d169d4013d2e387b7808065b3f8
Author: Imran Rashid <[email protected]>
Date: 2015-02-25T22:00:00Z
failing test case (though its crappy)
commit 29f0a8a10c685ea2742d239a748bc6c5d7798380
Author: Imran Rashid <[email protected]>
Date: 2015-02-27T19:19:12Z
fix use of LargeByteBuffer in some tests, create UploadPartialBlock
commit dcb46697d59fa77ac643e438b346eb28972d9e8f
Author: Imran Rashid <[email protected]>
Date: 2015-02-27T20:59:04Z
add real test case for uploading large blocks (failing now)
commit 660f5e362439d79d8dfd000a805be0ad5181106c
Author: Imran Rashid <[email protected]>
Date: 2015-02-28T02:57:32Z
flesh out NettyBlockTransfer#uploadBlock
commit 4c228a07173e06f8da449db17b878d220e14dea0
Author: Imran Rashid <[email protected]>
Date: 2015-02-28T03:14:13Z
minor cleanup
commit cf7c3a7067aaa61732782995984f17fa94a6cff7
Author: Imran Rashid <[email protected]>
Date: 2015-02-28T04:45:50Z
cleanup abandonded block uploads
commit fe90fd682d71ce4c156dde9e6a016e7923e65aad
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T15:46:19Z
crank up memory for tests
commit 857f3dfae649d56dbd37a022ddbe594c2a9bd0ac
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T15:47:16Z
fix LargeByteBuffer dispose()
commit b700723033d49094a4c0ee6f43a23592b29ae01f
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T16:55:19Z
maven needs you to request lots of extra memory for some reason
commit 6b102a028df0fae8d60b5cb2174fc4f380a81627
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T19:17:16Z
passing tests! assorted little changes
commit 84df2dd20664fa9333a2c01e06ed1ec1159e1b51
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T20:07:59Z
fix tests
commit 7a5d1acb4a512372800d7829aa676594c1f0eed3
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T20:12:30Z
dont kill jenkins with huge tests
commit 38d6993f7f687f6d857970229444644ddfe30db1
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T20:12:38Z
test cleanup
commit fc0d118b48d16a41f1e690b073c156fbf1a47437
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T20:25:58Z
fixup test & fix bug in WrappedLargeByteBuffer
commit 0e3700f3b69031bb91e6756d2abbbc993a3e8dca
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T21:12:55Z
cleanup
commit 6f6a8d7c512ab66ee8f03fa725d97533d0672c8e
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T21:17:24Z
cleanup
commit 9de88665fca9462f6ef6a395f0ad60a0275d1bd0
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T22:08:55Z
style fixes
commit ef8808555c2caddefbe5809e4c194bcc0675f40b
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T22:17:02Z
minor cleanup
commit 17d0c1a122e082e7263398fadaa75f2200544f04
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T22:30:31Z
get streaming to compile
commit f603ba4ae053e537514e8eb67e6cb5e3456578b0
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T22:38:04Z
very basic assertion in large broadcast test
commit cd84c6996c5fd48f2f9f2855e7f3e7a75ba5865a
Author: Imran Rashid <[email protected]>
Date: 2015-03-02T22:39:10Z
but turn test off b/c it requires large heap
commit 8c81f1c3e9bbd8f5ed477e033e8b351878fbe293
Author: Imran Rashid <[email protected]>
Date: 2015-03-24T01:49:39Z
Merge branch 'master' into SPARK-1391_2g_partition_limit
Conflicts:
core/src/main/scala/org/apache/spark/storage/BlockManager.scala
commit da8612405452693834bef90f85cbc81900987308
Author: Imran Rashid <[email protected]>
Date: 2015-03-26T15:18:51Z
rip out everything attempting to send > 2GB over the network; add some
exceptions; placeholders for some tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
