Re: MapReduce with Apache-Ignite
lalit wrote > You've mentioned that spilling task is worked upon. Does it mean in the > current version because of eviction we may loose mapper output? or the > node will crash complaining the out of memory error? In this case Ignite will consume all the available memory and crash with OOME. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MapReduce-with-Apache-Ignite-tp8007p8560.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: MapReduce with Apache-Ignite
Hi, Currently mapper output is stored in-memory only. We are working on spilling algorithm at the moment which will flush part of it to the disk according to some pre-configured thresholds. As per job logs, they are not supported at the moment by Ignite in the same way as it is done for Hadoop. We add similar functionality soon. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MapReduce-with-Apache-Ignite-tp8007p8314.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: MapReduce with Apache-Ignite
Regarding first point about mapper output - I intend to know that the mapper output will not be in-memory. Thus I/O operation on the local filesystem at mapper side still exists? Regarding second point - I tried running the job using the hadoop accelerator, I want to locate the logs for the job id, the mapper tasks & reducer tasks. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MapReduce-with-Apache-Ignite-tp8007p8018.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: MapReduce with Apache-Ignite
Hi, 1. Mapper output should be written to the same place as if it job was run through native Apache Hadoop engine. Apache Ignite Hadoop Accelerator can work without IGFS at all. 2. Currently you will not see jobs in Resource Manager because they are executed through separate engine. We will improve this in future. Please clarify what logs do you mean? Vladimir. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MapReduce-with-Apache-Ignite-tp8007p8016.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
