What happens if you set it to 100000?
On Oct 15, 2009, at 11:48 AM, Eric Lubow <[email protected]> wrote:
My connection section of the script is here:
# Connect to the database
my $socket = new Thrift::Socket('localhost',9160);
$socket->setSendTimeout(2500);
$socket->setRecvTimeout(7500);
my $transport = new Thrift::BufferedTransport($socket,2048,2048);
my $protocol = new Thrift::BinaryProtocol($transport);
my $client = Cassandra::CassandraClient->new($protocol);
I even tried it with combinations of 1024 as the size and 1000 as
the SendTimeout and 5000 as the RecvTimeout.
-e
On Thu, Oct 15, 2009 at 11:42 AM, Jake Luciani <[email protected]>
wrote:
I think it's 100ms. I need to increase it to match python I guess.
Sent from my iPhone
On Oct 15, 2009, at 11:40 AM, Jonathan Ellis <[email protected]>
wrote:
What is the default?
On Thu, Oct 15, 2009 at 10:37 AM, Jake Luciani <[email protected]>
wrote:
You need to call
$socket->setRecvTimeout()
With a higher number in ms.
On Oct 15, 2009, at 11:26 AM, Eric Lubow <[email protected]> wrote:
Using the Thrift Perl API into Cassandra, I am running into what is
endearingly referred to as the 4 bytes of doom:
TSocket: timed out reading 4 bytes from localhost:9160
The script I am using is fairly simple. I have a text file that has
about
3.6 million lines that are formatted like: [email protected] 1234
The Cassandra dataset is a single column family called Users in the
Mailings
keyspace with a data layout of:
Users = {
'[email protected]': {
email: '[email protected]',
person_id: '123456',
send_dates_2009-09-30: '2245',
send_dates_2009-10-01: '2247',
},
}
There are about 3.5 million rows in the Users column family and each
row has
no more than 4 columns (listed above). Some only have 3 (one of the
send_dates_YYYY-MM-DD isn't there).
The script parses it and then connects to Cassandra and does a
get_slice and
counts the return values adding that to a hash:
my ($value) = $client->get_slice(
'Mailings',
$email,
Cassandra::ColumnParent->new({
column_family => 'Users',
}),
Cassandra::SlicePredicate->new({
slice_range => Cassandra::SliceRange->new({
start => 'send_dates_2009-09-29',
finish => 'send_dates_2009-10-30',
}),
}),
Cassandra::ConsistencyLevel::ONE
);
$counter{($#{$value} + 1)}++;
For the most part, this script times out after 1 minute or so.
Replacing the
get_slice with a get_count, I can get it to about 2 million queries
before I
get the timeout. Replacing the get_slice with a get, I make it to
about 2.5
million before I get the timeout. The only way I could get it to
run all
the way through was to add a 1/100 of a second sleep during every
iteration.
I was able to get the script to complete when I shut down
everything else
on the machine (and it took 177m to complete). But since this is a
semi-production machine, I had to turn everything back on afterwards.
So for poops and laughs (at the recommendation of jbellis), I
rewrote the
script in Python and it has since run (using get_slice) 3 times fully
without timing out (approximately 130m in Python) with everything else
running on the machine.
My question is, having seen this same thing in the PHP API and it is
my
understanding that the Perl API was based on the PHP API,
could http://issues.apache.org/jira/browse/THRIFT-347 apply to Perl
here
too? Is anyone else seeing this issue? If so, have you gotten
around it?
Thanks.
-e
