Using Illustrate with Javascript UDF throws NEP
-----------------------------------------------
Key: PIG-2478
URL: https://issues.apache.org/jira/browse/PIG-2478
Project: Pig
Issue Type: Bug
Affects Versions: 0.11
Environment: Mac OSX Lion 10.7.2, Hadoop (0.20.205.0), Pig (Apache Pig
version 0.11.0-SNAPSHOT (r1230844).
Reporter: dan young
When I try to run illustrate while using a Javascript UDF, I always get a
NEP....but if I do a dump/store, everything works fine.
dans-MacBook-Pro:pig danoyoung$ pig -x local
2012-01-12 22:16:57,131 [main] INFO org.apache.pig.Main - Logging error
messages to: /Users/danoyoung/projects/pig/pig_1326431817128.log
2012-01-12 22:16:57,386 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: file:///
2012-01-12 22:16:57.483 java[27809:1903] Unable to load realm info from
SCDynamicStore
grunt> set io.sort.mb 500;
grunt> register '/Users/danoyoung/projects/pig/udf/udf.js' using
org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
2012-01-12 22:17:24,803 [main] INFO org.apache.pig.scripting.js.JsScriptEngine
- Register scripting UDF: get_date_marker
2012-01-12 22:17:24,804 [main] INFO org.apache.pig.scripting.js.JsScriptEngine
- Register scripting UDF: get_record
grunt> register '/usr/local/pig/piggybank.jar';
grunt>
grunt> a = LOAD
'/Users/danoyoung/Downloads/adwords_KeywordDailyReport_1-1-2012_daily' USING
org.apache.pig.piggybank.storage.XMLLoader('row') AS (doc:chararray);
grunt>
grunt> b = FOREACH a GENERATE FLATTEN(myfuncs.get_record(doc)) AS
(dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float);
grunt> describe b;
b: {dw_date_marker: int,ad_network_ad_group_key: long,ad_network_keyword_key:
long,firstpagecpc: int,qualityscore: int,cost: float,position: float}
grunt>
grunt> illustrate b;
2012-01-12 22:17:25,642 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: file:///
2012-01-12 22:17:25,846 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
2012-01-12 22:17:25,860 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-01-12 22:17:25,860 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-01-12 22:17:25,870 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig script settings are added to the job
2012-01-12 22:17:25,887 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-01-12 22:17:26,161 [main] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2012-01-12 22:17:26,721 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
2012-01-12 22:17:26,723 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-01-12 22:17:26,723 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-01-12 22:17:26,724 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig script settings are added to the job
2012-01-12 22:17:26,725 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
java.lang.NullPointerException
at org.apache.pig.scripting.js.JsFunction.exec(JsFunction.java:200)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:225)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:262)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194)
at
org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257)
at
org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238)
at
org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103)
at
org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98)
at
org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166)
at org.apache.pig.PigServer.getExamples(PigServer.java:1202)
at
org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:523)
at org.apache.pig.Main.main(Main.java:148)
2012-01-12 22:17:26,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2997: Encountered IOException. Exception : null
Details at logfile: /Users/danoyoung/projects/pig/pig_1326431817128.log
grunt>
Here's the log details:
Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. Exception : null
java.io.IOException: Exception : null
at org.apache.pig.PigServer.getExamples(PigServer.java:1208)
at
org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:523)
at org.apache.pig.Main.main(Main.java:148)
================================================================================
Here's the Javascript UDF:
get_record.outputSchema =
"dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float";
function get_record(doc){
var rec = new Object;
rec.dt = doc.substring(doc.search(/date=/ig)+6,doc.search(/\"
campaignid=/ig));
//rec.campaign_id =
doc.substring(doc.search(/campaignid=/ig)+12,doc.search(/\" adgroupid=/ig));
rec.ad_network_ad_group_key =
doc.substring(doc.search(/adgroupid=/ig)+11,doc.search(/\" keywordid=/ig));
rec.ad_network_keyword_key =
doc.substring(doc.search(/keywordid=/ig)+11,doc.search(/\" keyword=/ig));
rec.firstpagecpc =
doc.substring(doc.search(/firstpagecpc=/ig)+14,doc.search(/\"
qualityscore=/ig))/1000000;
rec.qualityscore =
doc.substring(doc.search(/qualityscore=/ig)+14,doc.search(/\" imps=/ig));
rec.cost = doc.substring(doc.search(/cost=/ig)+6,doc.search(/\"
pos=/ig))/1000000;
rec.position =
doc.substring(doc.search(/pos=/ig)+5,doc.search(/\"><\/row/ig));
rec.dw_date_marker = get_date_marker(rec.dt);
return {dw_date_marker:rec.dw_date_marker,
ad_network_ad_group_key:rec.ad_network_ad_group_key,
ad_network_keyword_key:rec.ad_network_keyword_key,
firstpagecpc:rec.firstpagecpc,
qualityscore:rec.qualityscore,
cost:rec.cost,
position:rec.position};
}
function get_date_marker(dt){
var dateMarkers = {
_2012_01_01:1204,
_2012_01_02:1205,
_2012_01_03:1206,
_2012_01_04:1207,
_2012_01_05:1208,
_2012_01_06:1209,
_2012_01_07:1210,
_2012_01_08:1211,
_2012_01_09:1212,
_2012_01_10:1213,
_2012_01_11:1214,
_2012_01_12:1215,
_2012_01_13:1216,
_2012_01_14:1217,
_2012_01_15:1218,
_2012_01_16:1219,
_2012_01_17:1220,
_2012_01_18:1221,
_2012_01_19:1222,
_2012_01_20:1223,
_2012_01_21:1224,
_2012_01_22:1225,
_2012_01_23:1226,
_2012_01_24:1227,
_2012_01_25:1228,
_2012_01_26:1229,
_2012_01_27:1230,
_2012_01_28:1231,
_2012_01_29:1232,
_2012_01_30:1233,
_2012_01_31:1234,
_2012_02_01:1235,
_2012_02_02:1236,
_2012_02_03:1237,
_2012_02_04:1238,
_2012_02_05:1239,
_2012_02_06:1240,
_2012_02_07:1241,
_2012_02_08:1242,
_2012_02_09:1243,
_2012_02_10:1244,
_2012_02_11:1245,
_2012_02_12:1246,
_2012_02_13:1247,
_2012_02_14:1248,
_2012_02_15:1249,
_2012_02_16:1250,
_2012_02_17:1251,
_2012_02_18:1252,
_2012_02_19:1253,
_2012_02_20:1254,
_2012_02_21:1255,
_2012_02_22:1256,
_2012_02_23:1257,
_2012_02_24:1258,
_2012_02_25:1259,
_2012_02_26:1260,
_2012_02_27:1261,
_2012_02_28:1262,
_2012_02_29:1263,
_2012_03_01:1264,
_2012_03_02:1265,
_2012_03_03:1266,
_2012_03_04:1267,
_2012_03_05:1268,
_2012_03_06:1269,
_2012_03_07:1270,
_2012_03_08:1271,
_2012_03_09:1272,
_2012_03_10:1273,
_2012_03_11:1274,
_2012_03_12:1275,
_2012_03_13:1276,
_2012_03_14:1277,
_2012_03_15:1278,
_2012_03_16:1279,
_2012_03_17:1280,
_2012_03_18:1281,
_2012_03_19:1282,
_2012_03_20:1283,
_2012_03_21:1284,
_2012_03_22:1285,
_2012_03_23:1286,
_2012_03_24:1287,
_2012_03_25:1288,
_2012_03_26:1289,
_2012_03_27:1290,
_2012_03_28:1291,
_2012_03_29:1292,
_2012_03_30:1293,
_2012_03_31:1294,
_2012_04_01:1295,
_2012_04_02:1296,
_2012_04_03:1297,
_2012_04_04:1298,
_2012_04_05:1299,
_2012_04_06:1300,
_2012_04_07:1301,
_2012_04_08:1302,
_2012_04_09:1303,
_2012_04_10:1304,
_2012_04_11:1305,
_2012_04_12:1306,
_2012_04_13:1307,
_2012_04_14:1308,
_2012_04_15:1309,
_2012_04_16:1310,
_2012_04_17:1311,
_2012_04_18:1312,
_2012_04_19:1313,
_2012_04_20:1314,
_2012_04_21:1315,
_2012_04_22:1316,
_2012_04_23:1317,
_2012_04_24:1318,
_2012_04_25:1319,
_2012_04_26:1320,
_2012_04_27:1321,
_2012_04_28:1322,
_2012_04_29:1323,
_2012_04_30:1324,
_2012_05_01:1325,
_2012_05_02:1326,
_2012_05_03:1327,
_2012_05_04:1328,
_2012_05_05:1329,
_2012_05_06:1330,
_2012_05_07:1331,
_2012_05_08:1332,
_2012_05_09:1333,
_2012_05_10:1334,
_2012_05_11:1335,
_2012_05_12:1336,
_2012_05_13:1337,
_2012_05_14:1338,
_2012_05_15:1339,
_2012_05_16:1340,
_2012_05_17:1341,
_2012_05_18:1342,
_2012_05_19:1343,
_2012_05_20:1344,
_2012_05_21:1345,
_2012_05_22:1346,
_2012_05_23:1347,
_2012_05_24:1348,
_2012_05_25:1349,
_2012_05_26:1350,
_2012_05_27:1351,
_2012_05_28:1352,
_2012_05_29:1353,
_2012_05_30:1354,
_2012_05_31:1355,
_2012_06_01:1356,
_2012_06_02:1357,
_2012_06_03:1358,
_2012_06_04:1359,
_2012_06_05:1360,
_2012_06_06:1361,
_2012_06_07:1362,
_2012_06_08:1363,
_2012_06_09:1364,
_2012_06_10:1365,
_2012_06_11:1366,
_2012_06_12:1367,
_2012_06_13:1368,
_2012_06_14:1369,
_2012_06_15:1370,
_2012_06_16:1371,
_2012_06_17:1372,
_2012_06_18:1373,
_2012_06_19:1374,
_2012_06_20:1375,
_2012_06_21:1376,
_2012_06_22:1377,
_2012_06_23:1378,
_2012_06_24:1379,
_2012_06_25:1380,
_2012_06_26:1381,
_2012_06_27:1382,
_2012_06_28:1383,
_2012_06_29:1384,
_2012_06_30:1385,
_2012_07_01:1386,
_2012_07_02:1387,
_2012_07_03:1388,
_2012_07_04:1389,
_2012_07_05:1390,
_2012_07_06:1391,
_2012_07_07:1392,
_2012_07_08:1393,
_2012_07_09:1394,
_2012_07_10:1395,
_2012_07_11:1396,
_2012_07_12:1397,
_2012_07_13:1398,
_2012_07_14:1399,
_2012_07_15:1400,
_2012_07_16:1401,
_2012_07_17:1402,
_2012_07_18:1403,
_2012_07_19:1404,
_2012_07_20:1405,
_2012_07_21:1406,
_2012_07_22:1407,
_2012_07_23:1408,
_2012_07_24:1409,
_2012_07_25:1410,
_2012_07_26:1411,
_2012_07_27:1412,
_2012_07_28:1413,
_2012_07_29:1414,
_2012_07_30:1415,
_2012_07_31:1416,
_2012_08_01:1417,
_2012_08_02:1418,
_2012_08_03:1419,
_2012_08_04:1420,
_2012_08_05:1421,
_2012_08_06:1422,
_2012_08_07:1423,
_2012_08_08:1424,
_2012_08_09:1425,
_2012_08_10:1426,
_2012_08_11:1427,
_2012_08_12:1428,
_2012_08_13:1429,
_2012_08_14:1430,
_2012_08_15:1431,
_2012_08_16:1432,
_2012_08_17:1433,
_2012_08_18:1434,
_2012_08_19:1435,
_2012_08_20:1436,
_2012_08_21:1437,
_2012_08_22:1438,
_2012_08_23:1439,
_2012_08_24:1440,
_2012_08_25:1441,
_2012_08_26:1442,
_2012_08_27:1443,
_2012_08_28:1444,
_2012_08_29:1445,
_2012_08_30:1446,
_2012_08_31:1447,
_2012_09_01:1448,
_2012_09_02:1449,
_2012_09_03:1450,
_2012_09_04:1451,
_2012_09_05:1452,
_2012_09_06:1453,
_2012_09_07:1454,
_2012_09_08:1455,
_2012_09_09:1456,
_2012_09_10:1457,
_2012_09_11:1458,
_2012_09_12:1459,
_2012_09_13:1460,
_2012_09_14:1461,
_2012_09_15:1462,
_2012_09_16:1463,
_2012_09_17:1464,
_2012_09_18:1465,
_2012_09_19:1466,
_2012_09_20:1467,
_2012_09_21:1468,
_2012_09_22:1469,
_2012_09_23:1470,
_2012_09_24:1471,
_2012_09_25:1472,
_2012_09_26:1473,
_2012_09_27:1474,
_2012_09_28:1475,
_2012_09_29:1476,
_2012_09_30:1477,
_2012_10_01:1478,
_2012_10_02:1479,
_2012_10_03:1480,
_2012_10_04:1481,
_2012_10_05:1482,
_2012_10_06:1483,
_2012_10_07:1484,
_2012_10_08:1485,
_2012_10_09:1486,
_2012_10_10:1487,
_2012_10_11:1488,
_2012_10_12:1489,
_2012_10_13:1490,
_2012_10_14:1491,
_2012_10_15:1492,
_2012_10_16:1493,
_2012_10_17:1494,
_2012_10_18:1495,
_2012_10_19:1496,
_2012_10_20:1497,
_2012_10_21:1498,
_2012_10_22:1499,
_2012_10_23:1500,
_2012_10_24:1501,
_2012_10_25:1502,
_2012_10_26:1503,
_2012_10_27:1504,
_2012_10_28:1505,
_2012_10_29:1506,
_2012_10_30:1507,
_2012_10_31:1508,
_2012_11_01:1509,
_2012_11_02:1510,
_2012_11_03:1511,
_2012_11_04:1512,
_2012_11_05:1513,
_2012_11_06:1514,
_2012_11_07:1515,
_2012_11_08:1516,
_2012_11_09:1517,
_2012_11_10:1518,
_2012_11_11:1519,
_2012_11_12:1520,
_2012_11_13:1521,
_2012_11_14:1522,
_2012_11_15:1523,
_2012_11_16:1524,
_2012_11_17:1525,
_2012_11_18:1526,
_2012_11_19:1527,
_2012_11_20:1528,
_2012_11_21:1529,
_2012_11_22:1530,
_2012_11_23:1531,
_2012_11_24:1532,
_2012_11_25:1533,
_2012_11_26:1534,
_2012_11_27:1535,
_2012_11_28:1536,
_2012_11_29:1537,
_2012_11_30:1538,
_2012_12_01:1539,
_2012_12_02:1540,
_2012_12_03:1541,
_2012_12_04:1542,
_2012_12_05:1543,
_2012_12_06:1544,
_2012_12_07:1545,
_2012_12_08:1546,
_2012_12_09:1547,
_2012_12_10:1548,
_2012_12_11:1549,
_2012_12_12:1550,
_2012_12_13:1551,
_2012_12_14:1552,
_2012_12_15:1553,
_2012_12_16:1554,
_2012_12_17:1555,
_2012_12_18:1556,
_2012_12_19:1557,
_2012_12_20:1558,
_2012_12_21:1559,
_2012_12_22:1560,
_2012_12_23:1561,
_2012_12_24:1562,
_2012_12_25:1563,
_2012_12_26:1564,
_2012_12_27:1565,
_2012_12_28:1566,
_2012_12_29:1567,
_2012_12_30:1568,
_2012_12_31:1569};
var rec = new Object;
rec.dt = dt.replace(/-/g, "_");
rec.date_marker = dateMarkers["_" + rec.dt];
return rec.date_marker;
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
