Without the fsync guarrantees are weakened a lot more than the fsync
lying case.
Also, you didn't mention the batch size on your avro sink that is
sending data to the avro-source. This is a major factor on your
throughput because each batch causes one sync. If you have big batches,
you'll have few fsyncs and significantly better performance.
I am weirded out by the fact that Danny is getting improved performance
by running multiple parallel file sinks... Are they each on separate
disks or something? I can't imagine what could cause a performance gain
if they were all on the same disk. Would likely expect more write head
skipping around and degradation even...
On 10/23/2012 03:31 PM, Jagadish Bihani wrote:
Hi Denny
Thanks for the inputs.
Btw when you say you tested another case without 'fsync'; I think
you changed the file channel code to comment out 'flush' part of it.
And if we rely on OS flushing then still it can be reasonably reliable.
Is that right?
Regards,
Jagadish
On 10/22/2012 07:08 PM, Denny Ye wrote:
hi Jagadish,
I have tested performance of FileChannel recently. Here I can
support the test report to you for your thinking and questions at
this thread.
Talking about the comparison between FileChannel and File Sink.
FileChannel supports both sequential writer and random reader, there
have so many times shift of magnetic head, it's slow than the
sequential writing much more.
'fsync' command has consuming much time than writing, almost
100times/sec, same as number mentioned from Brock. Also, I didn't
know why there have such difference between your two servers. I think
it might be related with OS version (usage between fsync and
fdatasync instruction) or disk driver (RAID, caching strategy, and so
on).
Throughput of single FileChannel is almost 3-5MB/sec in my
environment. Thus I used 5 channels with 18MB/sec. It's hard to
believe the linear increasing with more channels. Meanwhile, it look
like the limit of throughput with 'fsync' operation. I tested another
case without 'fsync' operation after each batch, almost
35-40MB/sec(Also, I removed the pre-allocation at disk writing in
this case).
Hope useful for you.
PS : I heard that OS has demon thread to flush page cache to
disk asynchronously with second latency, does it's effective for
amount of data with tolerant loss?
-Regards
Denny Ye
2012/10/22 Jagadish Bihani <[email protected]
<mailto:[email protected]>>
Hi
I am writing this on top of another thread where there was
discussion on "fsync lies" and
only file channel used fsync and not file sink. :
-- I tested the fsync performance on 2 machines (On 1 machine I
was getting very good throughput
using file channel and on another almost 100 times slower with
almost same hardware configuration.)
using following code
#define PAGESIZE 4096
int main(int argc, char *argv[])
{
char my_write_str[PAGESIZE];
char my_read_str[PAGESIZE];
char *read_filename= argv[1];
int readfd,writefd;
readfd = open(read_filename,O_RDONLY);
writefd = open("written_file",O_WRONLY|O_CREAT,777);
int len=lseek(readfd,0,2);
lseek(readfd,0,0);
int iterations = len/PAGESIZE;
int i;
struct timeval t0,t1;
for(i=0;i<iterations;i++)
{
read(readfd,my_read_str,PAGESIZE);
write(writefd,my_read_str,PAGESIZE);
*gettimeofday(&t0,0);**
** fsync(writefd);**
** gettimeofday(&t1,0);*
long elapsed = (t1.tv_sec-t0.tv_sec)*1000000 +
t1.tv_usec-t0.tv_usec;
printf("Elapsed time is= %ld \n",elapsed);
}
close(readfd);
close(writefd);
}
-- As expected it requires typically 50000 microseconds for fsync
to complete on one machine and 200 microseconds
on another machine it took 290 microseconds to complete on an
average. So is machine with higher
performance is doing a 'fsync lie'?
i
-- If I have understood it clearly; "fsync lie" means the data is
not actually written to disk and it is in
some disk/controller buffer. I) Now if disk loses power due to
some shutdown or any other disaster, data will
be lost. II) Can data be lost even without it ? (e.g. if it is
keeping data in some disk buffer and if fsync is being
invoked continuously then will that data can also be lost? If
only part -I is true; then it can be acceptable
because probability of shutdown is usually less in production
environment. But if even II is true then there is a
problem.
-- But on the machine where disk doesn't lie performance of flume
using File channel is very low (I have seen it
maximum 100 KB/sec even with sufficient DirectMemory
allocation.) Does anybody have stats about throughput
of file channel ? Is anybody getting better performance with file
channel (without fsync lies). What is the recommended
usage of it for an average scenario ? (Transferring files of few
MBs to HDFS sink continuously on typical hardware
(16 core processors, 16 GB RAM etc.)
Regards,
Jagadish
On 10/10/2012 11:30 PM, Brock Noland wrote:
Hi,
On Wed, Oct 10, 2012 at 11:22 AM, Jagadish Bihani
<[email protected]> <mailto:[email protected]>
wrote:
Hi Brock
I will surely look into 'fsync lies'.
But as per my experiments I think "file channel" is causing the issue.
Because on those 2 machines (one with higher throughput and other with
lower)
I did following experiment:
cat Source -memory channel - file sink.
Now with this setup I got same throughput on both the machines. (around 3
MB/sec)
Now as I have used "File sink" it should also do "fsync" at some point of
time.
'File Sink' and 'File Channel' both do disk writes.
So if there is differences in disk behaviour then even in the 'File Sink' it
should be visible.
Am I missing something here?
File sink does not call fsync.
Regards,
Jagadish
On 10/10/2012 09:35 PM, Brock Noland wrote:
OK your disk that is giving you 40KB/second is telling you the truth
and the faster disk is lying to you. Look up "fsync lies" to see what
I am referring to.
A spinning disk can do 100 fsync operations per second (this is done
at the end of every batch). That is how I estimated your event size,
40KB/second is doing 40KB / 100 = 409 bytes.
Once again, if you want increased performance, you should increase the
batch size.
Brock
On Wed, Oct 10, 2012 at 11:00 AM, Jagadish Bihani
<[email protected]> <mailto:[email protected]>
wrote:
Hi
Yes. It is around 480 - 500 bytes.
On 10/10/2012 09:24 PM, Brock Noland wrote:
How big are your events? Average about 400 bytes?
Brock
On Wed, Oct 10, 2012 at 5:11 AM, Jagadish Bihani
<[email protected]> <mailto:[email protected]>
wrote:
Hi
Thanks for the inputs Brock. After doing several experiments
eventually problem boiled down to disks.
-- But I had used the same configuration (so all software components
are
same in all 3 machines)
on all 3 machines.
-- In User guide it is written that if multiple file channel instances
are
active on the same agent then
different disks are preferable. But in my case only one file channel is
active per agent.
-- Only one pattern I observed that on the machines where I got better
performance have multiple disks.
But I don't understand how that will help if I have only 1 active file
channel.
-- What is the impact of the type of disk/disk device driver on
performance?
I mean I don't understand
with 1 disk I am getting 40 KB/sec and with other 2 MB/sec.
Could you please elaborate on File channel and disks correlation.
Regards,
Jagadish
On 10/09/2012 08:01 PM, Brock Noland wrote:
Hi,
Using file channel, in terms of performance, the number and type of
disks is going to be much more predictive of performance than CPU or
RAM. Note that consumer level drives/controllers will give you much
"better" performance because they lie to you about when your data is
actually written to the drive. If you search for "fsync lies" you'll
find more information on this.
You probably want to increase the batch size to get better performance.
Brock
On Tue, Oct 9, 2012 at 2:46 AM, Jagadish Bihani
<[email protected]> <mailto:[email protected]>
wrote:
Hi
My flume setup is:
Source Agent : cat source - File Channel - Avro Sink
Dest Agent : avro source - File Channel - HDFS Sink.
There is only 1 source agent and 1 destination agent.
I measure throughput as amount of data written to HDFS per second.
( I have rolling interval 30 sec; so If 60 MB file is generated in 30
sec
the
throughput is : -- 2 MB/sec ).
I have run source agent on various machines with different hardware
configurations :
(In all cases I run flume agent with JAVA OPTIONS as
"-DJAVA_OPTS="-Xms500m -Xmx1g -Dcom.sun.management.jmxremote
-XX:MaxDirectMemorySize=2g")
JDK is 32 bit.
Experiment 1:
=====
RAM : 16 GB
Processor: Intel Xeon E5620 @ 2.40 GHz (16 cores).
64 bit Processor with 64 bit Kernel.
Throughput: 2 MB/sec
Experiment 2:
======
RAM : 4 GB
Processor: Intel Xeon E5504 @ 2.00GHz (4 cores). 32 bit Processor
64 bit Processor with 32 bit Kernel.
Throughput : 30 KB/sec
Experiment 3:
======
RAM : 8 GB
Processor:Intel Xeon E5520 @ 2.27 GHz (16 cores).32 bit Processor
64 bit Processor with 32 bit Kernel.
Throughput : 80 KB/sec
-- So as can be seen there is huge difference in the throughput with
same
configuration but
different hardware.
-- In the first case where throughput is more RES is around 160 MB in
other
cases it is in
the range of 40 MB - 50 MB.
Can anybody please give insights that why there is this huge difference
in
the throughput?
What is the correlation between RAM and filechannel/HDFS sink
performance
and also
with 32-bit/64 bit kernel?
Regards,
Jagadish
