Just coming back to update you all on the solution. It was as David suggested just not letting varnish manipulate X-Forwarded-For, allowed haproxy -> varnish configuration to work for the app :D
Thanks for your help guys.. --- posted at http://www.serverphorums.com http://www.serverphorums.com/read.php?10,508289,515997#msg-515997