> HFile store wants more of an object store with full object reads and writes.
No, definitely not full object reads, we use HDFS positioned reads, which allow us to request, within a gigabyte plus store file, much smaller byte ranges (e.g. 64 KB), and receive back only the requested data. We can "seek" around the file. > > > Is this a reasonable thing to attempt? Or am I out of my mind? > > Check back in this mailing list archive for opinions on hbase atop s3. Aside from several IMHO showstopper performance problems, the shortest answer is HBase often wants to promptly read back store files it has written, and S3 is too eventual often enough (transient 404s or 500s) to preclude reliable operation. Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) ----- Original Message ----- > From: Jagane Sundar <[email protected]> > To: "[email protected]" <[email protected]> > Cc: > Sent: Wednesday, December 28, 2011 10:42 AM > Subject: RE: Writing the WAL to a different filesystem from the HFiles > > Hello Mikael, > > I am fairly new to HBase, so correct me if I am wrong - it seems to me that > the > WAL and the HFile store impose distinctly different requirements upon the > underlying storage. The WAL really wants Posix semantics (append, flush, > etc.) > in a distributed filesystem whereas the HFile store wants more of an object > store with full object reads and writes. > > Adding this configuration option, or better yet, making the WAL fully > pluggable > as described in https://issues.apache.org/jira/browse/HBASE-4529, opens up a > lot > more storage architectures to HBase. > > Thanks, > Jagane > ________________________________________ > From: Mikael Sitruk [[email protected]] > Sent: Wednesday, December 28, 2011 7:29 AM > To: [email protected] > Subject: Re: Writing the WAL to a different filesystem from the HFiles > > What is the advantage of such configuration? > Mikael.s > On Dec 28, 2011 1:10 AM, "Stack" <[email protected]> wrote: > >> On Mon, Dec 26, 2011 at 8:06 PM, Jagane Sundar <[email protected]> > wrote: >> > Is this a reasonable thing to attempt? Or am I out of my mind? >> > >> >> Check back in this mailing list archive for opinions on hbase atop s3. >> St.Ack >> >
