RE: Local sockets

2010-12-06 Thread Vladimir Rodionov
, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Saturday, December 04, 2010 3:04 PM To: dev@hbase.apache.org Subject: Re: Local sockets On Sat, Dec 4, 2010 at 2:57 PM, Vladimir Rodionov vrodio...@carrieriq.comwrote

Re: Local sockets

2010-12-06 Thread Todd Lipcon
From: Todd Lipcon [t...@cloudera.com] Sent: Saturday, December 04, 2010 3:04 PM To: dev@hbase.apache.org Subject: Re: Local sockets On Sat, Dec 4, 2010 at 2:57 PM, Vladimir Rodionov vrodio...@carrieriq.comwrote: From my own experiments performance difference is huge even

RE: Local sockets

2010-12-06 Thread Vladimir Rodionov
: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Monday, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Todd, The major

Re: Local sockets

2010-12-06 Thread Ryan Rawson
, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Monday, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio...@carrieriq.comwrote

Re: Local sockets

2010-12-06 Thread Jay Booth
regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Monday, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets

Re: Local sockets

2010-12-06 Thread Todd Lipcon
their results to any of the mailing lists! -Todd From: Todd Lipcon [t...@cloudera.com] Sent: Monday, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio

Re: Local sockets

2010-12-06 Thread Jay Booth
, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Todd, The major hdfs problem is inefficient processing of multiple streams in parallel - multiple readers/writers per one

Re: Local sockets

2010-12-06 Thread Ryan Rawson
10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Todd, The major hdfs problem is inefficient processing of multiple streams in parallel - multiple readers/writers per one physical drive result

Re: Local sockets

2010-12-06 Thread Jay Booth
, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Monday, December 06, 2010 10:04 AM To: dev@hbase.apache.org Subject: Re: Local sockets On Mon, Dec 6, 2010 at 9:59 AM, Vladimir Rodionov vrodio

Re: Local sockets

2010-12-06 Thread Rajappa Iyer
Ryan Rawson ryano...@gmail.com writes: So are we talking about re-implementing IO scheduling in Hadoop at the application level? Essentially yes. It does seem nasty to do that, but realistically there's no way that a general purpose I/O scheduler in the OS can schedule I/O optimally for M/R

RE: Local sockets

2010-12-06 Thread Vladimir Rodionov
Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Rajappa Iyer [...@panix.com] Sent: Monday, December 06, 2010 12:24 PM To: dev@hbase.apache.org Subject: Re: Local sockets Ryan Rawson ryano...@gmail.com writes: So are we talking about

Local sockets

2010-12-04 Thread Leen Toelen
Hi, has anyone tested the performance impact (when there is a hdfs datanode and a hbase node on the same machine) of using unix domain sockets communication or shared memory ipc using nio? I guess this should make a difference on reads? Regards, Leen

Re: Local sockets

2010-12-04 Thread Todd Lipcon
Hi Leen, Check out HDFS-347 for more info on this. I hope to pick this back up in 2011 - in 2010 we mostly focused on stability above performance in HBase's interactions with HDFS. Thanks -Todd On Sat, Dec 4, 2010 at 12:28 PM, Leen Toelen toe...@gmail.com wrote: Hi, has anyone tested the

RE: Local sockets

2010-12-04 Thread Vladimir Rodionov
-mail: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Saturday, December 04, 2010 12:30 PM To: dev@hbase.apache.org Subject: Re: Local sockets Hi Leen, Check out HDFS-347 for more info on this. I hope to pick this back up in 2011

Re: Local sockets

2010-12-04 Thread Todd Lipcon
Lipcon [t...@cloudera.com] Sent: Saturday, December 04, 2010 12:30 PM To: dev@hbase.apache.org Subject: Re: Local sockets Hi Leen, Check out HDFS-347 for more info on this. I hope to pick this back up in 2011 - in 2010 we mostly focused on stability above performance in HBase's interactions

Re: Local sockets

2010-12-04 Thread Ryan Rawson
: vrodio...@carrieriq.com From: Todd Lipcon [t...@cloudera.com] Sent: Saturday, December 04, 2010 12:30 PM To: dev@hbase.apache.org Subject: Re: Local sockets Hi Leen, Check out HDFS-347 for more info on this. I hope to pick this back up in 2011