[
https://issues.apache.org/jira/browse/PIG-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013125#comment-14013125
]
Cheolsoo Park edited comment on PIG-3967 at 5/30/14 12:08 AM:
--------------------------------------------------------------
[~daijy], thank you for the explanation. If that's the case, let's pass a
boolean param to {{LogicalPlan.validate()}}. So instead we can do the following-
{code}
diff --git a/src/org/apache/pig/PigServer.java
b/src/org/apache/pig/PigServer.java
index 0b2285e..d7ec664 100644
--- a/src/org/apache/pig/PigServer.java
+++ b/src/org/apache/pig/PigServer.java
@@ -1677,12 +1677,12 @@ public class PigServer {
}
}
- void validateQuery() throws FrontendException {
+ private void validateQuery() throws FrontendException {
String query = buildQuery();
QueryParserDriver parserDriver = new QueryParserDriver(
pigContext, scope, fileNameMap );
try {
LogicalPlan plan = parserDriver.parse( query );
- plan.validate(pigContext, scope);
+ plan.validate(pigContext, scope, true);
} catch(FrontendException ex) {
scriptCache.remove( scriptCache.size() -1 );
throw ex;
@@ -1741,7 +1741,7 @@ public class PigServer {
}
private void compile() throws IOException {
- lp.validate(pigContext, scope);
+ lp.validate(pigContext, scope, false);
currDAG.postProcess();
}
diff --git a/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
b/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
index dbbd5d1..b753121 100644
--- a/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
+++ b/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
@@ -165,8 +165,9 @@ public class LogicalPlan extends BaseOperatorPlan {
return Integer.toString(hos.getHashCode().asInt());
}
-
- public void validate(PigContext pigContext, String scope) throws
FrontendException {
+
+ public void validate(PigContext pigContext, String scope, boolean
skipInputOutputValidation)
+ throws FrontendException {
new DanglingNestedNodeRemover(this).visit();
new ColumnAliasConversionVisitor(this).visit();
@@ -204,7 +205,7 @@ public class LogicalPlan extends BaseOperatorPlan {
// compute whether output data is sorted or not
new SortInfoSetter(this).visit();
- if (!(pigContext.inExplain || pigContext.inDumpSchema)) {
+ if (!(skipInputOutputValidation || pigContext.inExplain ||
pigContext.inDumpSchema)) {
// Validate input/output file
new InputOutputFileValidatorVisitor(this, pigContext).visit();
}
{code}
My main concern is that if you set a boolean flag to true and false before and
after a method call, it will make someone wonder about the logic. It certainly
did to me.
I can upload a patch that includes my suggestions. Let me know.
was (Author: cheolsoo):
[~daijy], thank you for the explanation. If that's the case, let's a boolean
param to {{LogicalPlan.validate()}}. So instead we can do the following-
{code}
diff --git a/src/org/apache/pig/PigServer.java
b/src/org/apache/pig/PigServer.java
index 0b2285e..d7ec664 100644
--- a/src/org/apache/pig/PigServer.java
+++ b/src/org/apache/pig/PigServer.java
@@ -1677,12 +1677,12 @@ public class PigServer {
}
}
- void validateQuery() throws FrontendException {
+ private void validateQuery() throws FrontendException {
String query = buildQuery();
QueryParserDriver parserDriver = new QueryParserDriver(
pigContext, scope, fileNameMap );
try {
LogicalPlan plan = parserDriver.parse( query );
- plan.validate(pigContext, scope);
+ plan.validate(pigContext, scope, true);
} catch(FrontendException ex) {
scriptCache.remove( scriptCache.size() -1 );
throw ex;
@@ -1741,7 +1741,7 @@ public class PigServer {
}
private void compile() throws IOException {
- lp.validate(pigContext, scope);
+ lp.validate(pigContext, scope, false);
currDAG.postProcess();
}
diff --git a/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
b/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
index dbbd5d1..b753121 100644
--- a/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
+++ b/src/org/apache/pig/newplan/logical/relational/LogicalPlan.java
@@ -165,8 +165,9 @@ public class LogicalPlan extends BaseOperatorPlan {
return Integer.toString(hos.getHashCode().asInt());
}
-
- public void validate(PigContext pigContext, String scope) throws
FrontendException {
+
+ public void validate(PigContext pigContext, String scope, boolean
skipInputOutputValidation)
+ throws FrontendException {
new DanglingNestedNodeRemover(this).visit();
new ColumnAliasConversionVisitor(this).visit();
@@ -204,7 +205,7 @@ public class LogicalPlan extends BaseOperatorPlan {
// compute whether output data is sorted or not
new SortInfoSetter(this).visit();
- if (!(pigContext.inExplain || pigContext.inDumpSchema)) {
+ if (!(skipInputOutputValidation || pigContext.inExplain ||
pigContext.inDumpSchema)) {
// Validate input/output file
new InputOutputFileValidatorVisitor(this, pigContext).visit();
}
{code}
My main concern is that if you set a boolean flag to true and false before and
after a method call, it will make someone wonder about the logic. It certainly
did to me.
I can upload a patch that includes my suggestions. Let me know.
> Grunt fail if we running more statement after first store
> ---------------------------------------------------------
>
> Key: PIG-3967
> URL: https://issues.apache.org/jira/browse/PIG-3967
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.13.0
>
> Attachments: PIG-3967-1.patch, PIG-3967-2.patch
>
>
> We will hit output validation fail. The issue is caused by PIG-3545. We
> change PigServer.Graph.validateQuery() to invoke LogicalPlan.validate(),
> which will do the output validation. In Grunt mode, even after the first
> store, we will compile the entire statement cache, so the first store will be
> in the logical plan, validate on output fail.
> This makes datagenerator of Pigmix fail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
