Github user mortada commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173648674
@srowen I tested the python code in cluster mode (5 ec2 workers) and this
works fine
```
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block
manager 172.31.10.56:35937 with 6.6 GB RAM, BlockManagerId(4, 172.31.10.56,
35937)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block
manager 172.31.10.55:59871 with 6.6 GB RAM, BlockManagerId(0, 172.31.10.55,
59871)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block
manager 172.31.10.53:39162 with 6.6 GB RAM, BlockManagerId(1, 172.31.10.53,
39162)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block
manager 172.31.10.54:59145 with 6.6 GB RAM, BlockManagerId(2, 172.31.10.54,
59145)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block
manager 172.31.10.57:35000 with 6.6 GB RAM, BlockManagerId(3, 172.31.10.57,
35000)
In [1]: data = [1, 2, 3, 4, 5]
In [2]: counter = 0
In [3]: rdd = sc.parallelize(data)
In [4]: def increment_counter(x):
global counter
counter += x
...:
In [5]: rdd.foreach(increment_counter)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 172.31.10.55:59871 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 172.31.10.56:35937 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 172.31.10.57:35000 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 172.31.10.53:39162 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 172.31.10.54:59145 (size: 3.2 KB, free: 6.6 GB)
(other output skipped)
In [6]: print("Counter value: ", counter)
Counter value: 0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
