hadoop n00b asks, "Is adding more nodes the solution to such problem?"
Whatever else answers you get, you should append " ... and add more nodes." More nodes is never a bad thing ;-) --- Mark E. Sunderlin Solutions Architect |AOL Data Warehouse P: 703-256-6935 | C: 540-327-6222 AIM: MESunderlin 22000 AOL Way | Dulles, VA | 20166 From: hadoop n00b [mailto:[email protected]] Sent: Wednesday, March 16, 2011 3:33 AM To: [email protected] Subject: Fwd: Hadoop error 2 while joining two large tables Hello, I am trying to execute a query that joins two large tables (3 million and 20 million records). I am getting the Hadoop error code 2 during execution. This happens mainly while the reducers are running. Sometimes the reducers complete 100% and then the error comes. The logs talk about running out of Heap space and GC overhead limit exceeding. I am running a 6 node cluster with child JVM memory of 1GB. Are there any parameters I could tweak to make them run? Is adding more nodes the solution to such problem? Thanks!
