[jira] [Created] (MAPREDUCE-6769) Fix forgotten conversion from "slave" to "worker" in mapred script
Albert Chu created MAPREDUCE-6769: - Summary: Fix forgotten conversion from "slave" to "worker" in mapred script Key: MAPREDUCE-6769 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6769 Project: Hadoop Map/Reduce Issue Type: Bug Components: scripts Affects Versions: 3.0.0-alpha2 Reporter: Albert Chu Priority: Minor In HADOOP-13209 (commit 23c3ff85a9e73d8f0755e14f12cc7c89b72acddd), "slaves" was replaced with "workers" including the function name change from hadoop_common_slave_mode_execute to hadoop_common_worker_mode_execute and environment variable name change from HADOOP_SLAVE_MODE to HADOOP_WORKER_MODE. It appears this change was forgotten in hadoop-mapred-project/bin/mapred. Github pull request with fix to be sent shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Status: Patch Available (was: Open) Fix a checkstyle error from prior patch. Atleast 1 checkstyle error remains. It appears a style issue exists in ExampleDriver.java which is increased when staying consistent to the style in that file. > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624-branch-2.7.1.002.patch, MAPREDUCE-6624.001.patch, > MAPREDUCE-6624.002.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Attachment: MAPREDUCE-6624-branch-2.7.1.002.patch MAPREDUCE-6624.002.patch > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624-branch-2.7.1.002.patch, MAPREDUCE-6624.001.patch, > MAPREDUCE-6624.002.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Status: Open (was: Patch Available) > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624.001.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
Albert Chu created MAPREDUCE-6624: - Summary: Include TeraChecksum in Hadoop mapreduce examples driver Key: MAPREDUCE-6624 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 Project: Hadoop Map/Reduce Issue Type: Wish Components: examples Affects Versions: 2.7.1 Reporter: Albert Chu Priority: Minor In the mapreduce examples driver, you can run TeraGen, TeraSort, and TeraValidate. To my surprise, TeraChecksum wasn't included in the examples driver. I think it'd be nice to include. It can be used to compare the checksum of the input & output. It's worth noting that TeraValidate calculates the checksum as well. Patch to be posted shortly. I tested against 2.7.1 branch but couldn't against master. I am including patches for both. Patch also includes TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Status: Patch Available (was: Open) > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624.001.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Attachment: MAPREDUCE-6624-branch-2.7.1.001.patch MAPREDUCE-6624.001.patch > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624.001.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6281) Fix function comments in Terasort
[
https://issues.apache.org/jira/browse/MAPREDUCE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-6281:
--
Description:
Nothing fancy
{noformat}
- * @param job the job config
+ * @param conf the job config
{noformat}
probably leftover from the Hadoop 1.0 - 2.0 transition.
was:
Nothing fancy
- * @param job the job config
+ * @param conf the job config
probably leftover from the Hadoop 1.0 - 2.0 transition.
Fix function comments in Terasort
-
Key: MAPREDUCE-6281
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6281
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Albert Chu
Priority: Trivial
Nothing fancy
{noformat}
- * @param job the job config
+ * @param conf the job config
{noformat}
probably leftover from the Hadoop 1.0 - 2.0 transition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6281) Fix function comments in Terasort
Albert Chu created MAPREDUCE-6281: - Summary: Fix function comments in Terasort Key: MAPREDUCE-6281 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6281 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Albert Chu Priority: Trivial Nothing fancy - * @param job the job config + * @param conf the job config probably leftover from the Hadoop 1.0 - 2.0 transition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6281) Fix function comments in Terasort
[
https://issues.apache.org/jira/browse/MAPREDUCE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-6281:
--
Status: Patch Available (was: Open)
no tests added, trivial documentation fix
Fix function comments in Terasort
-
Key: MAPREDUCE-6281
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6281
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Albert Chu
Priority: Trivial
Attachments: MAPREDUCE-6281.001.patch
Nothing fancy
{noformat}
- * @param job the job config
+ * @param conf the job config
{noformat}
probably leftover from the Hadoop 1.0 - 2.0 transition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6281) Fix function comments in Terasort
[
https://issues.apache.org/jira/browse/MAPREDUCE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-6281:
--
Attachment: MAPREDUCE-6281.001.patch
Fix function comments in Terasort
-
Key: MAPREDUCE-6281
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6281
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Albert Chu
Priority: Trivial
Attachments: MAPREDUCE-6281.001.patch
Nothing fancy
{noformat}
- * @param job the job config
+ * @param conf the job config
{noformat}
probably leftover from the Hadoop 1.0 - 2.0 transition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367545#comment-14367545
]
Albert Chu commented on MAPREDUCE-5528:
---
Sure, I'll update the patch and retest just to make sure everything is fine and
dandy.
TeraSort fails with can't read paritions file - does not read partition
file from distributed cache
-
Key: MAPREDUCE-5528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 0.20.2, 3.0.0, 2.5.0, 2.4.1, 2.6.0
Reporter: Albert Chu
Assignee: Albert Chu
Priority: Minor
Attachments: MAPREDUCE-5528.patch
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the following error
when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was created
in the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output
directory or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is for
the filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the distributed
cache (I guess by default??).
But when I set things up with the networked file system and 'file://' scheme,
the working directory was the directory I was running my Hadoop binaries out
of.
The attached patch fixed things for me. It grabs the partition file from the
distributed cache all of the time, instead of trusting things underneath to
work out. It seems to be the right thing to do???
Apologies, I was unable to get this to reproduce under the TeraSort example
tests, such as TestTeraSort.java, so no test added. Not sure what the subtle
difference is in the setup. I tested under both HDFS 'file' scheme and the
patch worked under both.
--
[jira] [Updated] (MAPREDUCE-5050) Cannot find partition.lst in Terasort on Hadoop/Local File System
[
https://issues.apache.org/jira/browse/MAPREDUCE-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-5050:
--
Description:
I'm trying to simulate running Hadoop on Lustre by configuring it to use the
local file system using a single cloudera VM (cdh3u4).
I can generate the data just fine, but when running the sorting portion of the
program, I get an error about not being able to find the _partition.lst file.
It exists in the generated data directory.
Perusing the Terasort code, I see in the main method that has a Path reference
to partition.lst, which is created with the parent directory.
{noformat}
public int run(String[] args) throws Exception {
LOG.info(starting);
JobConf job = (JobConf) getConf();
Path inputDir = new Path(args[0]);
inputDir = inputDir.makeQualified(inputDir.getFileSystem(job));
Path partitionFile = new Path(inputDir, TeraInputFormat.PARTITION_FILENAME);
URI partitionUri = new URI(partitionFile.toString() +
# + TeraInputFormat.PARTITION_FILENAME);
TeraInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setJobName(TeraSort);
job.setJarByClass(TeraSort.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormat(TeraInputFormat.class);
job.setOutputFormat(TeraOutputFormat.class);
job.setPartitionerClass(TotalOrderPartitioner.class);
TeraInputFormat.writePartitionFile(job, partitionFile);
DistributedCache.addCacheFile(partitionUri, job);
DistributedCache.createSymlink(job);
job.setInt(dfs.replication, 1);
TeraOutputFormat.setFinalSync(job, true);
JobClient.runJob(job);
LOG.info(done);
return 0;
}
{noformat}
But in the configure method, the Path isn't created with the parent directory
reference.
{noformat}
public void configure(JobConf job) {
try {
FileSystem fs = FileSystem.getLocal(job);
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, job);
trie = buildTrie(splitPoints, 0, splitPoints.length, new Text(), 2);
} catch (IOException ie) {
throw new IllegalArgumentException(can't read paritions file, ie);
}
}
{noformat}
I modified the code as follows, and now sorting portion of the Terasort test
works using the
general file system. I think the above code is a bug.
{noformat}
public void configure(JobConf job) {
try {
FileSystem fs = FileSystem.getLocal(job);
Path[] inputPaths = TeraInputFormat.getInputPaths(job);
Path partFile = new Path(inputPaths[0],
TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, job);
trie = buildTrie(splitPoints, 0, splitPoints.length, new Text(), 2);
} catch (IOException ie) {
throw new IllegalArgumentException(can't read paritions file, ie);
}
}
{noformat}
was:
I'm trying to simulate running Hadoop on Lustre by configuring it to use the
local file system using a single cloudera VM (cdh3u4).
I can generate the data just fine, but when running the sorting portion of the
program, I get an error about not being able to find the _partition.lst file.
It exists in the generated data directory.
Perusing the Terasort code, I see in the main method that has a Path reference
to partition.lst, which is created with the parent directory.
public int run(String[] args) throws Exception {
LOG.info(starting);
JobConf job = (JobConf) getConf();
Path inputDir = new Path(args[0]);
inputDir = inputDir.makeQualified(inputDir.getFileSystem(job));
Path partitionFile = new Path(inputDir, TeraInputFormat.PARTITION_FILENAME);
URI partitionUri = new URI(partitionFile.toString() +
# + TeraInputFormat.PARTITION_FILENAME);
TeraInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setJobName(TeraSort);
job.setJarByClass(TeraSort.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormat(TeraInputFormat.class);
job.setOutputFormat(TeraOutputFormat.class);
job.setPartitionerClass(TotalOrderPartitioner.class);
TeraInputFormat.writePartitionFile(job, partitionFile);
DistributedCache.addCacheFile(partitionUri, job);
DistributedCache.createSymlink(job);
job.setInt(dfs.replication, 1);
TeraOutputFormat.setFinalSync(job, true);
JobClient.runJob(job);
LOG.info(done);
return 0;
}
But in the configure method, the Path isn't created with the parent directory
reference.
public void configure(JobConf job) {
try {
FileSystem fs =
[jira] [Commented] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368080#comment-14368080
]
Albert Chu commented on MAPREDUCE-5528:
---
Since I wrote my original patch, DistributedCache has been deprecated in favor
of using the job Context. Unfortunately, in TeraSort, there is no mapper or
reducer. All of the sorting is handled via the partitioner. As far as I can
tell, the Job context can't be accessed in the partitioner. B/c of that, this
really can't be handled through the patch I had before, assuming we don't want
to use deprecated code. Using the basic idea from [~ehiggs] in MAPREDUCE-5050
wouldn't have worked b/c the JobContext is again needed in newer versions of
FileOutputFormat.
I was trying to think of a clean way to do this but nothing came to mind each
of the ways I looked. I might just not see something that others would.
Open to suggestions.
TeraSort fails with can't read paritions file - does not read partition
file from distributed cache
-
Key: MAPREDUCE-5528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 0.20.2, 3.0.0, 2.5.0, 2.4.1, 2.6.0
Reporter: Albert Chu
Assignee: Albert Chu
Priority: Minor
Attachments: MAPREDUCE-5528.patch
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the following error
when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was created
in the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output
directory or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is for
the filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the distributed
cache (I
[jira] [Created] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
Albert Chu created MAPREDUCE-5528:
-
Summary: TeraSort fails with can't read paritions file - does
not read partition file from distributed cache
Key: MAPREDUCE-5528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Minor
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the
following error when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was
created in the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output
directory or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is
for the filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the
distributed cache (I guess by default??).
But when I set things up with the networked file system and 'file://'
scheme, the working directory was the directory I was running my
Hadoop binaries out of.
The attached patch fixed things for me. It grabs the partition file from the
distributed cache all of the time, instead of trusting things underneath to
work out. It seems to be the right thing to do???
Apologies, I was unable to get this to reproduce under the TeraSort
example tests, such as TestTeraSort.java, so no test added. Not sure what the
subtle difference is in the setup. I tested under both HDFS 'file' scheme
and the patch worked under both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-5528:
--
Attachment: MAPREDUCE-5528.patch
TeraSort fails with can't read paritions file - does not read partition
file from distributed cache
-
Key: MAPREDUCE-5528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Minor
Attachments: MAPREDUCE-5528.patch
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the following error
when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was created
in the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output
directory or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is for
the filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the distributed
cache (I guess by default??).
But when I set things up with the networked file system and 'file://' scheme,
the working directory was the directory I was running my Hadoop binaries out
of.
The attached patch fixed things for me. It grabs the partition file from the
distributed cache all of the time, instead of trusting things underneath to
work out. It seems to be the right thing to do???
Apologies, I was unable to get this to reproduce under the TeraSort example
tests, such as TestTeraSort.java, so no test added. Not sure what the subtle
difference is in the setup. I tested under both HDFS 'file' scheme and the
patch worked under both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Updated] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-5528:
--
Description:
I was trying to run TeraSort against a parallel networked file system, setting
things up via the 'file:// scheme. I always got the following error when
running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was created in
the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output directory
or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is for the
filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the distributed
cache (I guess by default??).
But when I set things up with the networked file system and 'file://' scheme,
the working directory was the directory I was running my Hadoop binaries out of.
The attached patch fixed things for me. It grabs the partition file from the
distributed cache all of the time, instead of trusting things underneath to
work out. It seems to be the right thing to do???
Apologies, I was unable to get this to reproduce under the TeraSort example
tests, such as TestTeraSort.java, so no test added. Not sure what the subtle
difference is in the setup. I tested under both HDFS 'file' scheme and the
patch worked under both.
was:
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the
following error when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at
[jira] [Created] (MAPREDUCE-5518) Fix typo can't read paritions file
Albert Chu created MAPREDUCE-5518:
-
Summary: Fix typo can't read paritions file
Key: MAPREDUCE-5518
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5518
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Trivial
Noticed a spelling error when I saw this error message
{noformat}
13/09/19 13:25:08 INFO mapreduce.Job: Task Id :
attempt_1379622083112_0002_m_000114_0, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
{noformat}
paritions should be partitions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5518) Fix typo can't read paritions file
[
https://issues.apache.org/jira/browse/MAPREDUCE-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Albert Chu updated MAPREDUCE-5518:
--
Attachment: MAPREDUCE-5518.patch
No tests added, it's a trivial typo fix.
Fix typo can't read paritions file
Key: MAPREDUCE-5518
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5518
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Trivial
Attachments: MAPREDUCE-5518.patch
Noticed a spelling error when I saw this error message
{noformat}
13/09/19 13:25:08 INFO mapreduce.Job: Task Id :
attempt_1379622083112_0002_m_000114_0, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
{noformat}
paritions should be partitions
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4902) Fix typo receievd should be received in log output
[ https://issues.apache.org/jira/browse/MAPREDUCE-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-4902: -- Status: Patch Available (was: Open) Fix typo receievd should be received in log output -- Key: MAPREDUCE-4902 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4902 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Albert Chu Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-4902.patch Noticed a typo in the log output, receievd should be received org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1356131733318_0002reduce=0map=attempt_1356131733318_0002_m_01_0,attempt_1356131733318_0002_m_03_0,attempt_1356131733318_0002_m_00_0 sent hash and receievd reply -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
