[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[
https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364291#comment-16364291
]
Steve Loughran commented on HADOOP-15227:
-
I'd done that for HDFS as that's always in my head. Mapred conf is the solution
here, though I may think about using a bit of reflection so if the jar isn't on
the CP, nothing fails.
> add mapreduce.outputcommitter.factory.scheme.s3a to core-default
>
>
> Key: HADOOP-15227
> URL: https://issues.apache.org/jira/browse/HADOOP-15227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Need to add this property to core-default.xml. It's documented as being
> there, but it isn't.
> {code}
>
> mapreduce.outputcommitter.factory.scheme.s3a
> org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> The committer factory to use when writing data to S3A filesystems.
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[
https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364275#comment-16364275
]
Jason Lowe commented on HADOOP-15227:
-
Yeah, mapred-default and mapred-site aren't loaded until the JobConf class is
loaded. A common mistake for code is to create a plain Configuration early in
{{main}} and try to lookup mapred properties (or even hdfs or yarn properties)
expecting to get the default if they are not set by the user. The easy fix is
to create a JobConf instead of a Configuration if the code knows it wants to do
mapred stuff.
> add mapreduce.outputcommitter.factory.scheme.s3a to core-default
>
>
> Key: HADOOP-15227
> URL: https://issues.apache.org/jira/browse/HADOOP-15227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Need to add this property to core-default.xml. It's documented as being
> there, but it isn't.
> {code}
>
> mapreduce.outputcommitter.factory.scheme.s3a
> org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> The committer factory to use when writing data to S3A filesystems.
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[
https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363783#comment-16363783
]
Steve Loughran commented on HADOOP-15227:
-
oh, maybe it's mapred default. Explains why I couldn't find it.
I'm implementing a filesystem diagnostics JAR in
https://github.com/steveloughran/cloudstore ; it wasn't finding it. But if it
doesn't get loaded until JobConf is loaded then yes, it'll be there.
In which case: docs are wrong. Let me fix them instead.
And I'll force load mapred-default & site through a bit of reflection if is
thsre
> add mapreduce.outputcommitter.factory.scheme.s3a to core-default
>
>
> Key: HADOOP-15227
> URL: https://issues.apache.org/jira/browse/HADOOP-15227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Need to add this property to core-default.xml. It's documented as being
> there, but it isn't.
> {code}
>
> mapreduce.outputcommitter.factory.scheme.s3a
> org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> The committer factory to use when writing data to S3A filesystems.
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15227) add mapreduce.outputcommitter.factory.scheme.s3a to core-default
[
https://issues.apache.org/jira/browse/HADOOP-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363017#comment-16363017
]
Jason Lowe commented on HADOOP-15227:
-
Does this go in core-default or mapred-default? The property name implies it
would not belong in core-default, and it currently has the proper value in
mapred-default. So maybe the documentation is what needs to be corrected
instead?
> add mapreduce.outputcommitter.factory.scheme.s3a to core-default
>
>
> Key: HADOOP-15227
> URL: https://issues.apache.org/jira/browse/HADOOP-15227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Need to add this property to core-default.xml. It's documented as being
> there, but it isn't.
> {code}
>
> mapreduce.outputcommitter.factory.scheme.s3a
> org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
>
> The committer factory to use when writing data to S3A filesystems.
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
