[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736990#action_12736990 ] Santhosh Srinivasan commented on PIG-889: - PigHadoopLogger implements the PigLogger interface. As part of the implementation it uses the Hadoop reporter for aggregating the warning messages. Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736091#action_12736091 ] Jeff Zhang commented on PIG-889: Santhosh, in my opinion if we want to support Counter in Pig, we only have to initialize the reporter in PigInputFormat. Because in map and reduce phase, we already have initialize the reporter. (PigMapReduce.java : Line 236 and PigMapBase: Line 217) Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736389#action_12736389 ] Jeff Zhang commented on PIG-889: Santhosh, you are right. I agree with you. The original purpose of PigHadoopLogger is not for Counter. We should not let the PigHadoopLogger to implement Counter for future extensibility. Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12736414#action_12736414 ] Jeff Zhang commented on PIG-889: Santhosh, BTW, since the PigHadoopLogger is not for Counter, why having the reporter object in it ? Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[
https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735129#action_12735129
]
Santhosh Srinivasan commented on PIG-889:
-
The issue here is the lack of support for counters within Pig.
The intention of warn method in the PigLogger interface was to allow sources
within Pig and UDFs for warning aggregation. Your use of the reporter within
the logger is not supported. An implementation detail prevents the correct use
of this interface for load functions. The Hadoop reporter object is provided in
the getRecordReader, map and reduce calls. For load functions, Pig provides an
interface and for UDFs, an abstract class. As a result, the logger instance
cannot be initialized in the loaders till we decide to add a method to support
it.
Will having the code from PigMapBase.map() in
PigInputFormat.java.getRecordReader work for you?
{code}
PigHadoopLogger pigHadoopLogger = PigHadoopLogger.getInstance();
pigHadoopLogger.setAggregate(aggregateWarning);
pigHadoopLogger.setReporter(reporter);
PhysicalOperator.setPigLogger(pigHadoopLogger);
{code}
Note that this is a workaround for your situation. I would highly recommend
that you move to the use of counters when they are supported.
Pig can not access reporter of PigHadoopLog in Load Func
Key: PIG-889
URL: https://issues.apache.org/jira/browse/PIG-889
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.4.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.4.0
Attachments: Pig_889_Patch.txt
I'd like to increment Counter in my own LoadFunc, but it will throw
NullPointerException. It seems that the reporter is not initialized.
I looked into this problem and find that it need to call
PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[
https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12733825#action_12733825
]
Santhosh Srinivasan commented on PIG-889:
-
Comments:
The reporter inside the logger is setup correctly in PigInputFormat for Hadoop.
However the usage of the logger to retrieve the reporter and then increment
counters is flawed for the following reasons:
1. In the test case, the new loader uses PigHadoopLogger directly. When the
loader is used in local mode, the notion of Hadoop disappears and the reference
to PigHadoopLogger is not usable (i.e., will result in a NullPointerException).
{code}
+ @Override
+ public Tuple getNext() throws IOException {
+ PigHadoopLogger.getInstance().getReporter().incrCounter(
+ MyCounter.TupleCounter, 1);
+ return super.getNext();
+ }
{code}
2. The loggers were meant for warning aggregations. Here, there is a case being
made to expand the capabilities to allow user defined counter aggregations. If
thats the case, then new methods have to be added to the PigLogger interface.
Pig can not access reporter of PigHadoopLog in Load Func
Key: PIG-889
URL: https://issues.apache.org/jira/browse/PIG-889
Project: Pig
Issue Type: Improvement
Components: impl
Affects Versions: 0.4.0
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Fix For: 0.4.0
Attachments: Pig_889_Patch.txt
I'd like to increment Counter in my own LoadFunc, but it will throw
NullPointerException. It seems that the reporter is not initialized.
I looked into this problem and find that it need to call
PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12732721#action_12732721 ] Hadoop QA commented on PIG-889: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413822/Pig_889_Patch.txt against trunk revision 794937. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/133/console This message is automatically generated. Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-889) Pig can not access reporter of PigHadoopLog in Load Func
[ https://issues.apache.org/jira/browse/PIG-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12732049#action_12732049 ] Hadoop QA commented on PIG-889: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413696/Pig_889_Patch.txt against trunk revision 793750. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/129/console This message is automatically generated. Pig can not access reporter of PigHadoopLog in Load Func Key: PIG-889 URL: https://issues.apache.org/jira/browse/PIG-889 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.4.0 Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.4.0 Attachments: Pig_889_Patch.txt I'd like to increment Counter in my own LoadFunc, but it will throw NullPointerException. It seems that the reporter is not initialized. I looked into this problem and find that it need to call PigHadoopLogger.getInstance().setReporter(reporter) in PigInputFormat. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
