Hi,
Thanks to your answer, we've found the problem. It was on reverse IP
resolution on the drivers we used (wrong configuration of the local
bind9). Apparently, not being able to reverse-resolve the IP address of
the nodes was the culprit of the 10s delay.
We've hit two other secondary problems with TorrentBroadcast though, in
case you're interested :
1 - Broadcasting a variable of about 2GB (1.8GB exactly) triggers a
"java.lang.OutOfMemoryError: Requested array size exceeds VM limit",
which is not the case with HttpBroadcast (I guess HttpBroadcast splits
the serialized variable in small chunks)
2 - Memory use of Torrent seems to be higher than Http (i.e. switching
from Http to Torrent triggers several OOM).
Additionally, a question : while HttpBroadcast stores the broadcast
pieces on disk (in spark.local.dir/spark-... ), TorrentBroadcast seems
not to use disk backend storage. Does it mean that HttpBroadcast can
handle bigger broadcast out of memory ? If so, it's too bad that this
design choice wasn't used for Torrent.
That being said, hats off to the people in charge of the broadcast
unloading wrt the lineage, this stuff works great !
Guillaume
Maybe there is a firewall issue that makes it slow for your nodes to connect through the IP
addresses they're configured with. I see there's this 10 second pause between "Updated info of
block broadcast_84_piece1" and "ensureFreeSpace(4194304) called" (where it actually
receives the block). HTTP broadcast used only HTTP fetches from the executors to the driver, but
TorrentBroadcast has connections between the executors themselves and between executors and the
driver over a different port. Where are you running your driver app and nodes?
Matei
On Oct 7, 2014, at 11:42 AM, Davies Liu <dav...@databricks.com> wrote:
Could you create a JIRA for it? maybe it's a regression after
https://issues.apache.org/jira/browse/SPARK-3119.
We will appreciate that if you could tell how to reproduce it.
On Mon, Oct 6, 2014 at 1:27 AM, Guillaume Pitel
<guillaume.pi...@exensa.com> wrote:
Hi,
I've had no answer to this on u...@spark.apache.org, so I post it on dev
before filing a JIRA (in case the problem or solution is already identified)
We've had some performance issues since switching to 1.1.0, and we finally
found the origin : TorrentBroadcast seems to be very slow in our setting
(and it became default with 1.1.0)
The logs of a 4MB variable with TorrentBroadcast : (15s)
14/10/01 15:47:13 INFO storage.MemoryStore: Block broadcast_84_piece1 stored
as bytes in memory (estimated size 171.6 KB, free 7.2 GB)
14/10/01 15:47:13 INFO storage.BlockManagerMaster: Updated info of block
broadcast_84_piece1
14/10/01 15:47:23 INFO storage.MemoryStore: ensureFreeSpace(4194304) called
with curMem=1401611984, maxMem=9168696115
14/10/01 15:47:23 INFO storage.MemoryStore: Block broadcast_84_piece0 stored
as bytes in memory (estimated size 4.0 MB, free 7.2 GB)
14/10/01 15:47:23 INFO storage.BlockManagerMaster: Updated info of block
broadcast_84_piece0
14/10/01 15:47:23 INFO broadcast.TorrentBroadcast: Reading broadcast
variable 84 took 15.202260006 s
14/10/01 15:47:23 INFO storage.MemoryStore: ensureFreeSpace(4371392) called
with curMem=1405806288, maxMem=9168696115
14/10/01 15:47:23 INFO storage.MemoryStore: Block broadcast_84 stored as
values in memory (estimated size 4.2 MB, free 7.2 GB)
(notice that a 10s lag happens after the "Updated info of block
broadcast_..." and before the MemoryStore log
And with HttpBroadcast (0.3s):
14/10/01 16:05:58 INFO broadcast.HttpBroadcast: Started reading broadcast
variable 147
14/10/01 16:05:58 INFO storage.MemoryStore: ensureFreeSpace(4369376) called
with curMem=1373493232, maxMem=9168696115
14/10/01 16:05:58 INFO storage.MemoryStore: Block broadcast_147 stored as
values in memory (estimated size 4.2 MB, free 7.3 GB)
14/10/01 16:05:58 INFO broadcast.HttpBroadcast: Reading broadcast variable
147 took 0.320907112 s 14/10/01 16:05:58 INFO storage.BlockManager: Found
block broadcast_147 locally
Since Torrent is supposed to perform much better than Http, we suspect a
configuration error from our side, but are unable to pin it down. Does
someone have any idea of the origin of the problem ?
For now we're sticking with the HttpBroadcast workaround.
Guillaume
--
Guillaume PITEL, Président
+33(0)626 222 431
eXenSa S.A.S.
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)184 163 677 / Fax +33(0)972 283 705
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
