[
https://issues.apache.org/jira/browse/PIG-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874871#comment-13874871
]
Rohini Palaniswamy commented on PIG-3672:
-----------------------------------------
Few comments:
- WeightedRangePartitioner.java and MapredUtil.java
Please do not add custom file formats. You can iterate through the
configuration and if the key starts with "fs." and ends with ".impl" set that
in the new configuration so that it is generic. Wondering how this was working
before for s3.
- QueryParserUtils.java
a) Can you add the Utils.hasFileSystemImpl check for the scheme before
adding to remote host. hbase:// currently is skipped because there is no host.
But this would future proof for other schemes that pig might support in future
which has host as well but no filesystem implementation.
b) If har is the scheme, then you need to return hdfs:// in the remote host
- JobControlCompiler.java
!fileuri.getScheme().equals("file:///") condition is wrong. Even if file://
we should write to output directory. You can just change
if (!outputPathString.contains("://") ||
outputPathString.startsWith("hdfs://")) {
to
if (Utils.hasFileSystemImpl(new Path(outputPathString), conf)) {
That should take care of fixing PIG-1424 as well.
> pig should not hardcode "hdfs://" path in code, should be configurable to
> other file system implementations
> -----------------------------------------------------------------------------------------------------------
>
> Key: PIG-3672
> URL: https://issues.apache.org/jira/browse/PIG-3672
> Project: Pig
> Issue Type: Bug
> Components: data, parser
> Affects Versions: 0.10.0, 0.12.0, 0.11.1
> Reporter: Suhas Satish
> Assignee: Suhas Satish
> Attachments: PIG-3672-1.patch, PIG-3672.patch
>
>
> QueryParserUtils.java has the code -
> result.add("hdfs://"+thisHost+":"+uri.getPort());
> I propose to make it generic like -
> result.add(uri.getScheme() + "://"+thisHost+":"+uri.getPort());
> Similarly jobControlCompiler.java has -
> if (!outputPathString.contains("://") ||
> outputPathString.startsWith("hdfs://")) {
> I have a patch version which I ran passing unit tests on. Will be uploading
> it shortly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
