[
https://issues.apache.org/jira/browse/HADOOP-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806095#action_12806095
]
Todd Lipcon commented on HADOOP-6513:
-------------------------------------
Rob: can you please post this test as a patch that just adds the new test case
and doesn't comment out or modify other parts of TestSequenceFile? You can
generate a patch using "svn diff". It's difficult to review what the test is
doing when you've uploaded the entire modified file. See
http://wiki.apache.org/hadoop/HowToContribute
As for the other JIRA, please keep discussion on each issue within the comments
for that issue. One of the central foundations of the Apache process is open
discussion and the ability to see how decisions were made. If you put comments
in this JIRA that pertain to another, people may not see them, and down the
road they certainly won't find them.
> SequenceFile.Sorter design issue and class-check bug
> -----------------------------------------------------
>
> Key: HADOOP-6513
> URL: https://issues.apache.org/jira/browse/HADOOP-6513
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: hadoop 20.1, java 1.6.0_17,fedora
> Reporter: robert Cook
> Attachments: TestSequenceFile.java
>
>
> SequenceFile.Writer takes key/value classes as creation arguments and checks
> for validity on every append.
> Reader does not take class arguments on creation because they are derived
> from the input file.
> Sorter takes key/value classes as creation arguments?? no point. should be
> derived from input.
> In any case, SortPass does not compare Sorter key/value classes with input
> file classes.
> No error is given for the following:
> private static void writeTest4(FileSystem fs, int count, int seed,
> Path file,
> SequenceFile.CompressionType compressionType,
> CompressionCodec codec, Configuration conf)
> throws IOException {
> fs.delete(file, true);
> LOG.info("creating " + count + " records with " + compressionType +
> " compression");
> SequenceFile.Writer writer =
> SequenceFile.createWriter(fs, conf, file,
> StringWritable.class, FloatWritable.class,
> compressionType, codec);
> FloatWritable x=new FloatWritable();
> StringWritable y=new StringWritable();
> for (int i = count-1; i >= 0; i--) {
> x.set(i); y.set(""+i);
> writer.append(y, x);
> }
> writer.close();
> }
> private static void sortTest(FileSystem fs, int count, int megabytes,
> int factor, boolean fast, Path file, Configuration
> conf)
> throws IOException {
> fs.delete(new Path(file+".sorted"), true);
> SequenceFile.Sorter sorter = newSorter(fs, fast, megabytes,
> factor, conf);
> LOG.debug("sorting " + count + " records");
> sorter.sort(file, file.suffix(".sorted"));
> LOG.info("done sorting " + count + " debug");
> }
>
> private static SequenceFile.Sorter newSorter(FileSystem fs,
> boolean fast,
> int megabytes, int factor, Configuration conf) {
> SequenceFile.Sorter sorter =
> fast
> ? new SequenceFile.Sorter(fs, new
> IntWritable.Comparator(),
> FloatWritable.class,
> IntWritable.class, conf)
> : new SequenceFile.Sorter(fs, FloatWritable.class,
> IntWritable.class, conf);
> sorter.setMemory(megabytes * 1024*1024);
> sorter.setFactor(factor);
> return sorter;
> }
> ---------------------Note String/Float does not match Float/Int
> Macintosh-2:datanode bobcook$ od -c file
> 0000000 S E Q 006 016 S t r i n g W r i t a
> 0000020 b l e \r F l o a t W r i t a b l
> 0000040 e \0 \0 \0 \0 \0 \0 203 ` n E J z 272 d 352
> 0000060 w 177 373 \n 364 M 276 \0 \0 \0 \n \0 \0 \0 006 \0
> 0000100 \0 \0 001 \0 4 @ 200 \0 \0 \0 \0 \0 \n \0 \0 \0
> 0000120 006 \0 \0 \0 001 \0 3 @ @ \0 \0 \0 \0 \0 \n \0
> 0000140 \0 \0 006 \0 \0 \0 001 \0 2 @ \0 \0 \0 \0 \0 \0
> 0000160 \n \0 \0 \0 006 \0 \0 \0 001 \0 1 ? 200 \0 \0 \0
> 0000200 \0 \0 \n \0 \0 \0 006 \0 \0 \0 001 \0 0 \0 \0 \0
> *
> 0000220
> Macintosh-2:datanode bobcook$ od -c file.sorted
> 0000000 S E Q 006 \r F l o a t W r i t a b
> 0000020 l e \v I n t W r i t a b l e \0 \0
> 0000040 \0 \0 \0 \0 6 364 343 \r 256 h U 222 365 T 7 l
> 0000060 357 i ~ } \0 \0 \0 \n \0 \0 \0 006 \0 \0 \0 001
> 0000100 \0 4 @ 200 \0 \0 \0 \0 \0 \n \0 \0 \0 006 \0 \0
> 0000120 \0 001 \0 3 @ @ \0 \0 \0 \0 \0 \n \0 \0 \0 006
> 0000140 \0 \0 \0 001 \0 2 @ \0 \0 \0 \0 \0 \0 \n \0 \0
> 0000160 \0 006 \0 \0 \0 001 \0 1 ? 200 \0 \0 \0 \0 \0 \n
> 0000200 \0 \0 \0 006 \0 \0 \0 001 \0 0 \0 \0 \0 \0
> 0000216
> NOTE OUTPUT FILE IS TOTALLY TOASTED, but no error was generated!
> PS: Your evaluation of my previous bug reports was enlightening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
