RE: listdir() python function is not wokring on hadoop

[Yigitbasi, Nezih]

I can call listdir to read from local filesystem in a python UDF. Did you 
implement your function as a proper UDF?
________________________________________
From: Haider [haider.n...@gmail.com]
Sent: Monday, December 02, 2013 5:22 AM
To: user@pig.apache.org
Subject: listdir() python function is not wokring on hadoop

Hi all

   is there any one who successfully used listdir() function to retrieve
files one by one from HDFS using python script.


 if __name__ == '__main__':

    for filename in os.listdir("/user/hdmaster/XML2"):
    print filename

ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks
exceeded allowed limit. FailedCount: 1. LastFailedTask:
task_201312020139_0025_m_000000
13/12/02 05:20:50 INFO streaming.StreamJob: killJob...

My intention is to take files one by one to parse.

Any help or suggestion on this will be so much helpful to me

Thanks
Haider