Eric, Thanks for that hint. Do you have a sense (or is this written somewhere) as to what tests may not work? I am running the tests as a user with very high permissions so I think they can write anywhere in HDFS, but I can double check that easily enough.
If there are tests that are known not to work, I can avoid those tests or at least not post so much to the mailing list about such tests. If you don't know, I'll just carry on and post what I learn. Obvious script bugs I can easily fix. Other more subtle issues require more investigation. Keys ________________________________ Keys Botzum Senior Principal Technologist WW Systems Engineering [email protected] 443-718-0098 MapR Technologies http://www.mapr.com On Apr 8, 2012, at 10:23 AM, Eric Newton wrote: > The auto tests are a bit flakey, especially when they rely on local > conditions, like the speed at which my workstation will run something. If > you have found a bug in one of the scripts, please make a Jira ticket. > > Right now several of the tests assume they have superuser rights to scribble > all over the distributed file system, see ACCUMULO-355. > > -Eric > > On Sun, Apr 8, 2012 at 10:01 AM, Keys Botzum <[email protected]> wrote: > Billie, > > I apologize for taking so long to reply to your message. As soon as I read it > I thought to myself "this is a very good argument." I did some more research > myself and talked with folks at MapR and we agreed that the right thing to do > here is to change MapR to be tolerant of a seek to position just after the > last byte. > > I've delayed posting this because I wanted to report the results of the > change and retest. This was delayed for reasons that have nothing to do with > Accumulo or really MapR, I just was too busy. > > In any case, we've tried the change in MapR (being tracked as change 6539) > and this does indeed address the issue below. The test case in question now > passes. Thank you VERY much for your help. It is much appreciated. > > I am continuing testing and have encountered additional issues but I haven't > yet had time to analyze them. I will post something when I have a more > coherent question. I would just like to confirm that all of the auto tests do > run successfully on stock Apache Hadoop? I ask because at least one of the > tests appears to have a bug in the script that runs it. I just want to make > sure I'm testing the right stuff. > > Incidentally, I have upgrade to the official Accumulo 1.4.0 release. > > > Keys > > > On Fri, Mar 30, 2012 at 11:15 PM, Billie J Rinaldi > <[email protected]> wrote: > It is my understanding that the file length is a valid file position. If you > are at position 0 and read one byte, you get the first byte of the file. If > you are at position length-1 and read one byte, you get the last byte of the > file. At that point you are at position equal to the file length, and > attempting to read an additional byte will give an EOF indicator (for an > InputStream read(), this would mean returning -1). Seeking to the file > length should just cause a read to give an EOF. > > Billie > > > ----- Original Message ----- > > From: "Keys Botzum" <[email protected]> > > To: [email protected] > > Sent: Friday, March 30, 2012 10:12:00 PM > > Subject: Re: Accumulo on MapR Continued > > Billie, > > > > > > Thanks for the quick reply. I did some checking and the Hadoop docs > > indicate that seeking past the end of the file is illegal. > > > > > > > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#seek%28long%29 > > seek > > public void seek (long desired) > > throws IOException Description copied from interface: Seekable Seek to > > the given offset from the start of the file. The next read() will be > > from that location. Can't seek past the end of the file. > > > > Specified by: seek in interface Seekable Throws: IOException > > > > > > The code below that is seeking to the file length is actually seeking > > one byte past the end of the file since the first byte in a file is > > byte zero. To me this seems like an Accumulo bug. Can this be fixed? > > > > > > Thanks, > > Keys > > ________________________________ > > Keys Botzum > > Senior Principal Technologist > > WW Systems Engineering > > [email protected] > > 443-718-0098 > > MapR Technologies > > http://www.mapr.com > > > > > > > > > > > > On Mar 30, 2012, at 5:04 PM, Billie J Rinaldi wrote: > > > > > > > > I ran the following JUnit test and verified that the value printed out > > was the file length. > > > > @Test > > public void testSeekToEOF() throws IOException { > > FileSystem fs = FileSystem.get(new Configuration()); > > FSDataInputStream fdi = fs.open(new Path("README")); > > System.out.println("seeking to end: " + fdi.available()); > > fdi.seek(fdi.available()); > > fdi.close(); > > } > > > > > > ----- Original Message ----- > > > > > > From: "Keys Botzum" < [email protected] > > > > > > > To: [email protected] > > > > > > Sent: Friday, March 30, 2012 4:56:09 PM > > > > > > Subject: Re: Accumulo on MapR Continued > > > > > > Can you share the program? I can try running it and answer your > > > > > > question. I can also of course check with MapR engineering to see if > > > > > > that's the issue. > > > > > > > > > > > > > > > > > > Much appreciated, > > > > > > Keys > > > > > > > > > > > > ________________________________ > > > > > > Keys Botzum > > > > > > Senior Principal Technologist > > > > > > WW Systems Engineering > > > > > > [email protected] > > > > > > 443-718-0098 > > > > > > MapR Technologies > > > > > > http://www.mapr.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 30, 2012, at 4:48 PM, Billie J Rinaldi wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Friday, March 30, 2012 2:07:49 PM, "Keys Botzum" < > > > > > > [email protected] > wrote: > > > > > > > > > > > > > > > > > > 30 11:00:15,778 [tabletserver.Tablet] ERROR: Failed to find lastkey > > > > > > > > > > > > > > > > > > Seeking beyond EOF, filelen 733, wantpos 733 > > > > > > > > > > > > It looks like Accumulo is trying to seek to the last position in a > > > > > > file. I just wrote a little program that opens a file in HDFS and > > > > > > seeks to its file length, and it didn't throw an error. Does MapR not > > > > > > allow this? If that is the desired MapR behavior, we could try to > > > > > > determine whether Accumulo could avoid performing that seek. > > > > > > > > > > > > Billie > > > > -- > ________________________________ > Keys Botzum > Senior Principal Technologist > WW Systems Engineering > [email protected] > 443-718-0098 > MapR Technologies > http://www.mapr.com > >
