http://www.compileonline.com/execute_python_online.php
#!/usr/local/bin/python2.7
print "Hello World!\n";
def ptile(value):
unique = set(value)
maps = {}
pc = float(1)/(len(unique)-1)
for n, i in enumerate(unique):
maps[i] = (n*pc)
return [maps.get(i) for i in value]
print ptile(10)
Hello World!
Traceback (most recent call last):
File "main.py", line 15, in
print ptile(10)
File "main.py", line 8, in ptile
unique = set(value)
TypeError: 'int' object is not iterable
Would it help?
2014-03-16 18:29 GMT+04:00 Jason W <[email protected]>:
> Serega, I tried this and it gave me output as expected i.e 0.0.
>
> Any idea what could be wrong with the UDF?
>
> @outputSchema("y:float")
> def ptile(value):
> unique = set(value)
> maps = {}
> pc = float(1)/(len(unique)-1)
> for n, i in enumerate(unique):
> maps[i] = (n*pc)
> return [maps.get(i) for i in value]
>
>
>
>
>
>
> On Sunday, March 16, 2014 4:46 AM, Serega Sheypak <
> [email protected]> wrote:
> Strange stacktrace. Do you get if from map task log of a job which reads
> the data?
>
> Try to replace your udf function with:
> @outputSchema("y:float")
> def ptile(value):
> return 0.0
>
> Just to see that your UDF is correct.
>
>
>
> 2014-03-16 3:09 GMT+04:00 Jason W <[email protected]>:
>
> > The stack trace I posted is from the logs. Thanks.
> >
> >
> >
> >
> >
> > On Saturday, March 15, 2014 6:42 PM, Ankit Bhatnagar <
> [email protected]>
> > wrote:
> > Check the log file ..it should in the folder where your are running the
> > script
> >
> >
> > On 3/15/14 2:12 PM, "Jason W" <[email protected]> wrote:
> >
> > >
> > >
> > >Updated for a typo:
> > >
> > >hi all,
> > >
> > >My script basically performs a calculation and outputs an array of
> values.
> > >
> > >
> > >Pig script:
> > >
> > >register pcent.py using jython as pc;
> > >data = load 'Data.csv' as (value:int);
> > >B = FOREACH data GENERATE pc.ptile(value) as ptiles;
> > >C = group B by (ptiles);
> > >store C into 'C';
> > >
> > >
> > >UDF:
> > >
> > >@outputSchema("y:float")
> > >def ptile(value):
> > > unique = set(value)
> > > maps = {}
> > > pc = float(1)/(len(unique)-1)
> > > for n, i in enumerate(unique):
> > > maps[i] = (n*pc)
> > > return [maps.get(i) for i in value]
> > >
> > >
> > >The python script runs fine but I keep getting an error in Pig and the
> > >stack trace is not very helpful:
> > >
> > >
> > >Backend error message
> > >---------------------
> > >org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> > >executing function
> > > at
> >
> >org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:12
> > >0)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:337)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:410)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
> > >or.getNext(PhysicalOperator.java:344)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.processPlan(POForEach.java:372)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.getNext(POForEach.java:297)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.runPipeline(PigGenericMapBase.java:283)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.map(PigGenericMapBase.java:278)
> > > at o
> > >
> > >Pig Stack Trace
> > >---------------
> > >ERROR 0: Error executing function
> > >
> > >org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> > >executing function
> > > at
> >
> >org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:12
> > >0)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:337)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:410)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
> > >or.getNext(PhysicalOperator.java:344)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.processPlan(POForEach.java:372)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.getNext(POForEach.java:297)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.runPipeline(PigGenericMapBase.java:283)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.map(PigGenericMapBase.java:278)
> >
> >==========================================================================
> > >======
> > >
> > >
> > >On Saturday, March 15, 2014 5:08 PM, Jason W <[email protected]> wrote:
> > >hi all,
> > >
> > >My script basically performs a calculation and outputs an array of
> values.
> > >
> > >
> > >Pig script:
> > >
> > >register pcent.py using jython as pc;
> > >data = load 'Data.csv' as (value:int);
> > >B = FOREACH data GENERATE pc.ptile(value);
> > >C = group B by (percentiles);
> > >store C into 'C';
> > >
> > >
> > >UDF:
> > >
> > >@outputSchema("y:float")
> > >def ptile(value):
> > > unique = set(value)
> > > maps = {}
> > > pc = float(1)/(len(unique)-1)
> > > for n, i in enumerate(unique):
> > > maps[i] = (n*pc)
> > > return [maps.get(i) for i in value]
> > >
> > >
> > >The python script runs fine but I keep getting an error in Pig and the
> > >stack trace is not very helpful:
> > >
> > >
> > >Backend error message
> > >---------------------
> > >org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> > >executing function
> > > at
> >
> >org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:12
> > >0)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:337)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:410)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
> > >or.getNext(PhysicalOperator.java:344)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.processPlan(POForEach.java:372)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.getNext(POForEach.java:297)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.runPipeline(PigGenericMapBase.java:283)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.map(PigGenericMapBase.java:278)
> > > at o
> > >
> > >Pig Stack Trace
> > >---------------
> > >ERROR 0: Error executing function
> > >
> > >org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> > >executing function
> > > at
> >
> >org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:12
> > >0)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:337)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
> > >ators.POUserFunc.getNext(POUserFunc.java:410)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
> > >or.getNext(PhysicalOperator.java:344)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.processPlan(POForEach.java:372)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
> > >ators.POForEach.getNext(POForEach.java:297)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.runPipeline(PigGenericMapBase.java:283)
> > > at
> >
> >org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
> > >Base.map(PigGenericMapBase.java:278)
> >
> >==========================================================================
> > >======
> > >
> >
> >
>
>
