Hi, community, I have limited number of machines, and with different size of RAM and different cores number. Is there any advice for how to imitate a real cluster with these small number of nodes?
Is the following way ok? 1st machine: 8 cores, 16G memory, 5 spark workers, one core per worker, 3GB mm per worker 2nd machine: 2 cores, 4G memory, 1 spark worker, one core per worker, 3GB mm per worker 3rd machine: 2 cores, 4G memory, 1 spark worker, one core per worker, 3GB mm per worker 4th machine: 4 cores, 16G memory. 3 spark worker, one core per worker, 3GB mm per worker And one Executor per Worker (since one Worker only has one core). If you have any other convenient way to get a cluster with at least 20 large memory nodes, please let me know, and by the way, I have zero experience about Amazon EC2, so I am a little scared about the potential monetary cost, please correct me if I am wrong. thanks, dachuan.
