We have deployed a pilot for making scribe log directly into a hdfs 0.19
cluster. We will be pushing these changes to scribe to the open-source
location this week. Also, there are a few changes to HDFS that were required
to make this work smoothly, e.g. HADOOP-2757. These changes are being
debated to figure out what will be pushed into hdfs trunk.
Also, here are some of the configuration settings:
<property>
<name>ipc.client.idlethreshold</name>
<value>10000</value>
<description>Defines the threshold number of connections
after which
connections will be inspected for idleness.
</description>
</property>
<property>
<name>ipc.client.connection.maxidletime</name>
<value>10000</value>
<description>The maximum time in msec after which a client will
bring down the
connection to the server.
</description>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>2</value>
<description>Indicates the number of retries a client will make to
establish
a server connection.
</description>
</property>
<property>
<name>ipc.server.listen.queue.size</name>
<value>128</value>
<description>Indicates the length of the listen queue for servers
accepting
client connections.
</description>
</property>
<property>
<name>ipc.server.tcpnodelay</name>
<value>true</value>
<description>Turn on/off Nagle's algorithm for the TCP socket
connection on
the server. Setting to true disables the algorithm and may
decrease latency
with a cost of more/smaller packets.
</description>
</property>
<property>
<name>ipc.client.tcpnodelay</name>
<value>true</value>
<description>Turn on/off Nagle's algorithm for the TCP socket
connection on
the client. Setting to true disables the algorithm and may
decrease latency
with a cost of more/smaller packets.
</description>
</property>
<property>
<name>ipc.ping.interval</name>
<value>5000</value>
<description>The Client sends a ping message to server every
period. This is helpful
to detect socket connections that were idle and have been
terminated by a failed server.
</description>
</property>
<property>
<name>ipc.client.connect.maxwaittime</name>
<value>5000</value>
<description>The Client waits for this much time for a socket
connect call to be establised
with the server.
</description>
</property>
<property>
<name>dfs.datanode.socket.write.timeout</name>
<value>20000</value>
<description>The dfs Client waits for this much time for a socket
write call to the datanode.
</description>
</property>
<property>
<name>dfs.leaserenewal.timeout</name>
<value>10000</value>
<description>The dfs writer waits for this much time after the
last successful lease renewal
before aborting the write to the file.
</description>
</property>
On Mon, May 18, 2009 at 12:22 AM, Amr Awadallah <[email protected]> wrote:
> Scribe from Facebook:
>
> http://developers.facebook.com/scribe/
>
> -- amr
>
>> All,
>>
>> Does anyone have a working method to capture system and application
>> logging for remote servers directly into HDFS in real-time?
>>
>> We have tested a few methods but all seem reliant on a batch process run
>> every 5 to 10 minutes.
>>
>> Any ideas would be appreciated?
>>
>>
>>
>> Warm Regards,
>>
>>
>> Scott
>>
>> ________________________________
>> Disclaimer: This email and any attachments thereto are confidential and it
>> may contain information protected by intellectual Property Laws or otherwise
>> legally privileged. It is intended solely for the use of the addressee(s).
>> If you are not the intended recipient, please notify the sender immediately
>> and delete this email and any attachments from your system and destroy any
>> printout thereof. Any unauthorized use or disclosure of this email and its
>> attachments is strictly prohibited and may be unlawful. MEEZA QSTP-LLC
>> (MEEZA) does not certify that this email or any attachments are free of
>> viruses or defects and recommends that you undertake your own virus checking
>> procedures before opening this e-mail or any attachments. MEEZA does not
>> accept any liability for any loss or damage resulting from the use of this
>> e-mail or any attachments. The contents of this e-mail and any attachments
>> do not necessarily represent the views or polices of MEEZA. As the integrity
>> of e-mails across the internet cannot be guaranteed, MEEZA does not accept
>> any liability for any e-mails and attachments which may violate the relevant
>> applicable laws.
>>
>>
>>
>>
>
>
>
