[
https://issues.apache.org/jira/browse/PIG-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531449#comment-13531449
]
Cheolsoo Park commented on PIG-2553:
------------------------------------
Hi Prashant,
Thanks for your responses:
# Agreed.
# Thanks.
# On a second thought, how about simplifying it even further?
{code}
if
("true".equals(pigContext.getProperties().getProperty(PIG_LOCATION_CHECK_STRICT)))
{
checkDuplicateStoreLoc(storeOps);
}
...
/**
* This method checks whether the multiple sinks (STORE) use the same
* "file-based" location. If yes, throws a runtime exception.
*
* @param storeOps
*/
private void checkDuplicateStoreLoc(Set<LOStore> storeOps) {
Set<String> uniqueStoreLoc = new HashSet<String>();
for(LOStore store : storeOps) {
String filename = store.getFileSpec().getFileName();
if(!uniqueStoreLoc.add(filename) &&
UriUtil.isHDFSFileOrLocalOrS3N(filename))
throw new RuntimeException("Script contains 2 or more STORE
statements writing to same location : "+ filename);
}
}
{code}
# Sure. That sounds reasonable. But can you add the new property to
{{pig.properties}} as well? I like to have a single place where all properties
are listed. As far as I know, {{pig.properties}} is only such a place as of now.
# I can't build {{admin.xml}}. I get the following error when running {{ant
docs}}:
{code}
[exec]
/home/cheolsoo/workspace/pig/src/docs/src/documentation/content/xdocs/admin.xml:33:66:
Element type "b" must be declared.
[exec]
/home/cheolsoo/workspace/pig/src/docs/src/documentation/content/xdocs/admin.xml:33:194:
The content of element type "p" must match
"(strong|em|code|sub|sup|br|img|icon|acronym|map|xi:include|a)"
{code}
Replacing {{<b></b>}} with {{<strong></strong>}} works for me. Also, it would
be nice if you could avoid using tabs for indentation. :-)
> Pig shouldn't allow attempts to write multiple relations into same directory
> ----------------------------------------------------------------------------
>
> Key: PIG-2553
> URL: https://issues.apache.org/jira/browse/PIG-2553
> Project: Pig
> Issue Type: Improvement
> Reporter: Dmitriy V. Ryaboy
> Assignee: Prashant Kommireddi
> Attachments: PIG-2553_1.patch, PIG-2553.patch
>
>
> We've seen multiple occasions where users accidentally try to store 2 or more
> different relations to the same destination directory. Currently, this passes
> the Pig planner and fails on MR side due to concurrent attempts to create the
> same part file on the reducer. This is extremely confusing to the user, and
> hard to debug.
> We should instead fail their scripts before they are even submitted, since we
> can identify the erroneous condition from the beginning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
