[
https://issues.apache.org/jira/browse/PIG-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705537#comment-13705537
]
Sergey commented on PIG-3376:
-----------------------------
OMG, it was so stupid. I had to click view ALL logs in jobtracker UI. My
Java code is called. I see that null pointer is there
Looks like my big data is not properly filtered.
I had such assumption and wrapped my Java code invocation code with
try/except
@outputSchema("centerLon:double, centerLat:double")
def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
print lon, lat, maxDist, angleWidth, azimuth
try:
spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
float(maxDist), float(angleWidth), float(azimuth))
return spatialCoord.longitude, spatialCoord.latitude
except BaseException, e:
print str(e)
return 1.0, 1.0
but failure happened. I see it in logs now (i was dummy before and didn't
click view ALL logs):
A line with bad value goes here
*47.4111 43.0136 675.0 0.0 55.0*
A lot of hadoop info goes here...
gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[34,19],gsmCellFilteredGrp[33,21]
C: cellUniqueLacCid[34,19],gsmCellFilteredGrp[33,21] R:
cellUniqueLacCid[34,19],gsmCellFixed[39,15],gsmCellProj[61,14]
2013-07-11 10:41:25,248 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and
reduceRetainSize=-1
2013-07-11 10:41:25,250 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:oozie (auth:SIMPLE)
cause:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
executing function
2013-07-11 10:41:25,250 WARN org.apache.hadoop.mapred.Child: Error running
child
org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
executing function
at
org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: Traceback (most recent call last):
File
"/data/disk0/mapred/local/taskTracker/oozie/distcache/-2997063519790422007_-650099178_1282707185/nameservice1/applications/kyc-analytics/gsmcellpreparator/workflows/pig/udf/geoSpatialUdfs.py",
line 12, in calculateDropShapedCenter
File
"/data/disk0/mapred/local/taskTracker/oozie/distcache/-2997063519790422007_-650099178_1282707185/nameservice1/applications/kyc-analytics/gsmcellpreparator/workflows/pig/udf/geoSpatialUdfs.py",
line 12, in calculateDropShapedCenter
at
mycompany.kyc.common.geo.model.builder.GeometryUtil.calculatePolarRadiusForDropShapredSector(GeometryUtil.java:76)
at
mycompany.kyc.common.geo.model.builder.GeometryBuilder.buildDropShapedSector(GeometryBuilder.java:198)
at
mycompany.kyc.common.geo.model.builder.GeometryBuilder.buildGeometry(GeometryBuilder.java:128)
at
mycompany.kyc.common.geo.model.builder.GeometryBuilder.getCentroid(GeometryBuilder.java:147)
at
mycompany.kyc.common.geo.model.builder.GeometryBuilder$getCentroid.call(Unknown
Source)
at
mycompany.pig.geo.GSMCellCalculator.getDropShapeCenter(GSMCellCalculator.groovy:49)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
The problem is in wrong hands :(
2013/7/11 serega.sheypak <[email protected]>
> Jython UDF invocation failure when importing java class in Cluster mode
> -----------------------------------------------------------------------
>
> Key: PIG-3376
> URL: https://issues.apache.org/jira/browse/PIG-3376
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11
> Environment: local, cluster
> Reporter: Sergey
>
> Hi, I'm testing my scripts in local mode then I run them in production using
> oozie.
> Locally everything works fine. My pig version is 0.11
> When I run the same script in cluster mode, I do get exception on line where
> jython udf is invoked. Here is my UDF, see it imports java class. This class
> is IN runtime, I give 100%:
> {code}
> from ru.pig.geo import GSMCellCalculator
> @outputSchema("centerLon:double, centerLat:double")
> def calculateDropShapedCenter(lon, lat, maxDist, angleWidth, azimuth):
> print lon, lat, maxDist, angleWidth, azimuth
> spatialCoord = GSMCellCalculator.getDropShapeCenter(lon, lat,
> float(maxDist), float(angleWidth), float(azimuth))
> return spatialCoord.longitude, spatialCoord.latitude
> {code}
> Here is a part of script:
> {code}
> register '$geoSpatialUdfs' using jython as udf;
> /*
> some code goes here....
> */
> gsmCellProj = FOREACH gsmCellFixed GENERATE
> branchId,
> cellId, lac,
> lon, lat,
> (int)azimuth, (int)midDist,
> (int)maxDist,
> cellType, (int)angWidth, gen,
> startAng,
> angWidthFixed, startAngFixed,
>
> FLATTEN(udf.calculateDropShapedCenter(lon, lat, midDist, angWidth, azimuth));
> {code}
> Here is the log STDOUT:
> {code}
> -07-10 17:33:30,729 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2013-07-10 17:33:30,729 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2013-07-10 17:33:30,737 [JobControl] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2013-07-10 17:33:31,562 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,562 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Processing aliases
> cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Processing aliases
> cellUniqueLacCid,gsmCell,gsmCellFiltered,gsmCellFilteredGrp,gsmCellFixed,gsmCellProj
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - detailed locations: M:
> gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
> C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
> cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - detailed locations: M:
> gsmCell[3,10],gsmCell[-1,-1],gsmCellFiltered[20,18],cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21]
> C: cellUniqueLacCid[33,19],gsmCellFilteredGrp[32,21] R:
> cellUniqueLacCid[33,19],gsmCellFixed[38,15],gsmCellProj[60,14]
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> 2013-07-10 17:33:31,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://prod-node016.lol.ru:50030/jobdetails.jsp?jobid=job_201307101220_0154
> Heart beat
> 2013-07-10 17:33:51,196 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 14% complete
> 2013-07-10 17:33:51,196 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 14% complete
> 2013-07-10 17:33:54,210 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2013-07-10 17:33:54,210 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2013-07-10 17:33:57,229 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 50% complete
> 2013-07-10 17:33:57,229 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 50% complete
> 2013-07-10 17:34:13,307 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:13,307 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:34:31,899 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:31,899 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:34:49,480 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:34:49,480 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> 2013-07-10 17:35:08,576 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 83% complete
> Heart beat
> 2013-07-10 17:35:26,170 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
> stop immediately on failure.
> 2013-07-10 17:35:26,170 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
> stop immediately on failure.
> 2013-07-10 17:35:26,170 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201307101220_0154 has failed! Stop running all dependent jobs
> 2013-07-10 17:35:26,170 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201307101220_0154 has failed!
> {code}
> Here is the log with error:
> {code}
> org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error
> executing function
> at
> org.apache.pig.scripting.jython.JythonFunction.exec(JythonFunction.java:120)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:376)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:354)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackage
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
