[
https://issues.apache.org/jira/browse/PIG-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875475#comment-13875475
]
Suhas Satish commented on PIG-3672:
-----------------------------------
* Removed custom file formats from WeightedRangePartitioner.java and
MapredUtil.java as I realized this was not really required and was not being
used anywhere.
* QueryParserUtils.java
** a) Do you mean to add the check at the code block shown in the diff below ?
The problem with this is that there is no access to Configuration object in
QueryParserUtils.java. Do you have recommendations on how and where to
instantiate a conf object to pass into Utils.hasFileSystemImpl as an argument?
{quote}
String scheme = uri.getScheme();
- if (scheme!=null && scheme.toLowerCase().equals("hdfs")||sc
+ if (scheme!=null && Utils.hasFileSystemImpl(p,conf)) {
{quote}
** b) Is this currently being returned correctly for "har" in trunk? If so,
this change would handle it right?
{quote}
if (scheme!=null && Utils.hasFileSystemImpl(p,conf)) {
if (uri.getHost()==null)
continue;
String thisHost = uri.getHost().toLowerCase();
if (scheme.toLowerCase().equals("har")) {
if (thisHost.startsWith(HAR_PREFIX)) {
thisHost = thisHost.substring(HAR_PREFIX.length());
}
scheme="hdfs";
}
if (!uri.getHost().isEmpty() &&
!thisHost.equals(defaultHost)) {
if (uri.getPort()!=-1)
result.add(scheme +
"://"+thisHost+":"+uri.getPort());
else
result.add(scheme + "://"+thisHost);
}
}
{quote}
* JobControlCompiler.java
This has been taken care of as -
{quote}
String outputPathString = st.getSFile().getFileName();
if (Utils.hasFileSystemImpl(new Path(outputPathString), conf)) {
conf.set("pig.streaming.log.dir",
new Path(outputPathString, LOG_DIR).toString());
} else {
{quote}
> pig should not hardcode "hdfs://" path in code, should be configurable to
> other file system implementations
> -----------------------------------------------------------------------------------------------------------
>
> Key: PIG-3672
> URL: https://issues.apache.org/jira/browse/PIG-3672
> Project: Pig
> Issue Type: Bug
> Components: data, parser
> Affects Versions: 0.10.0, 0.12.0, 0.11.1
> Reporter: Suhas Satish
> Assignee: Suhas Satish
> Attachments: PIG-3672-1.patch, PIG-3672.patch
>
>
> QueryParserUtils.java has the code -
> result.add("hdfs://"+thisHost+":"+uri.getPort());
> I propose to make it generic like -
> result.add(uri.getScheme() + "://"+thisHost+":"+uri.getPort());
> Similarly jobControlCompiler.java has -
> if (!outputPathString.contains("://") ||
> outputPathString.startsWith("hdfs://")) {
> I have a patch version which I ran passing unit tests on. Will be uploading
> it shortly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
