Nabeel wrote:
Kapila,
With your latest changes you always read the file in one go (i.e.
not in chunks) and you don't need a loop. The attached compact code
does the same thing.
Suggestion:
When we need to attach *very* large file, we may not be keep the
whole file in memory. IMO, this where chunk reading is really useful.
If you are to implement this, you'll have to change the code from
data_handler -> mime_body_part -> mime_output to om_output so that
you'll keep only chunk_size amount of bytes in memory at any given time.
Chunking is a very good idea, in fact that is the perfect solution for
large attachments.
However, AFAIK, we have chunking capability only on the client side
transport, not on the server side transport.
Hence, we got to implement chunking capability into simple axis server
transport first. (May be we can raise a Jira on this). Or we have to see
if we can piggyback on Apache for chunking.
Secondly, we got to research and find out the ideal chunk size. As we
have seen through the results sent by Kapila, too small a chunk size
makes too large file transmission very slow and too large chunk size
makes small file transmission very slow. Best is to not use chunking for
small files and use chunking, with a reasonable chunk size, for large
files. So we have several parameters to figure out.
i. Max size we can read a file in one go without chunking.
ii. When chunking is turned on, ideal chunk size.
As I mentioned earlier, we need some research around this to decide on
the figures.
Thanks,
Samisa...
-Nabeel
Samisa Abeysinghe wrote:
Cool, so there sure is an improvement when using stat on read side of
data handler. Hence lets use this solution.
Still we need to figure out what goes wrong with large files beyond 1M
Samisa...
Kapila Dissanayake wrote:
Hi
I did some modification to the data handler. Insted of using fixed
buffer size to read files
'stat' is used.
Please see the perfomance details.
Thank You
Kapila
Environment ---------- Processor - Intel
Celeron M 1.40GHz
RAM - 256 MB OS - Fedora Core 4. Java Server -
axis2 Java C Server - axis2 mtom C and java test
clients were used (in axis2 c data_handler.c, different buffer
sizes were used to read files)
Time Taken in milli seconds
File Size (kb) Java Server (ms) Java Server
(ms) C Server (ms) C Server (ms)
Java Client (ms) C Client (ms) C
Client (ms) Java Client (ms)
*Buffer Size 1024 * *Buffer
Size 1024 X 1024 * *Using stat* * * *Buffer Size '1024 *
*Buffer Size '1024 X 1024 * *Using stat *
----------------- ----------------- -----------------
----------------- ----------------- -----------------
----------------- ----------------- -----------------
----------------- ----------------- -----------------
-----------------
13.8 147 158 73 1884
78 188 143 136
28.2 515 488 78 1912
78 222 413 364
37.8 538 694 131 1917
81 232 474 377
132 571 807 212 2177
102 263 420 381
154 547 672 239 2215
117 347 389 383
165 688 683 397 2380
205 364 400 393
561 1340 1024 995 2537
777 637 899 775
762 1766 1336 1903 2890
1458 1248 1218 1051
1162 3464 1970 4859 3602
3405 1702 1718 1712
1761 4666 3274 10928 8028
7927 2780 3249 2691
2081 6777 2993 15066 11008
10864 3595 3281 2859
4241 23038 Could not transfer (Java client
Broken) 52853 33351 32957 16234
115777 7991
On 5/10/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Well, from this I am certain that we should not change the
buffer size
to 1024*1024, rather keep it at 1024.
Please try with stat, using a dynamic buffer size based on the file
size, my gut feel is that it is going to be the best solution.
Thanks,
Samisa...
Kapila Dissanayake wrote:
>
> Hi,
>
> I did some testing agian with to measure the binary file transfer
> times. In axis2 c data_handler.c, different buffer sizes (1024
and
> 1024*1024) were used to read files.
>
> I think using fixed size buffers to read binary files in axis2-C
has
> been affected in some cases to slow down the transfer.
>
> Thanks
>
> Kapila
>
> Please see the following test results
>
> Environment
>
>
>
>
>
>
> ----------
>
>
>
>
>
>
>
> Processor - Intel Celeron M 1.40GHz
>
>
>
>
> RAM - 256 MB
>
>
>
>
>
>
> OS - Fedora Core 4.
>
>
>
>
>
>
> Java Server - axis2 Java
>
>
>
>
>
> C Server - axis2
>
>
>
>
>
>
> mtom C and java test clients were used
>
>
>
> Time Taken in milli seconds
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> File Size (kb) Java Server (ms) Java
Server (ms) C
> Server (ms) C Server (ms)
> Java Client (ms) C
Client (ms) C Client (ms) Java
> Client (ms)
> *Buffer Size*
*1024 * *1024 X 1024* * * *1024*
> *1024 X 1024 *
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> ----------------- ----------------- -----------------
> 13.8 147 158
73 1884 188 143
> 28.2 515 488
78 1912 222 413
> 37.8 538 694 131
1917 232 474
> 132 571 807 212
2177 263 420
> 154 547 672 239
2215 347 389
> 165 688 683 397
2380 364 400
> 561 1340 1024 995
2537 637 899
> 762 1766 1336
1903 2890 1248 1218
>
1162 3464 1970 4859
3602 1702 1718
> 1761 4666 3274 10928
8028 2780 3249
> 2081 6777 2993 15066
11008 3595 3281
> 4241 23038 Could not transfer (Java client
Broken)
> 52853 33351 16234 115777
>
>
>
>
> On 5/8/06, *Samisa Abeysinghe* <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
> <mailto: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
>
> Hmmm, Interesting. There seem to be a problem in C client in
case of
> large attachments.
> This must be due to the fact that we are using a loop to
read the
> file.
> If we stat and read the file at once, there could be an
improvement.
>
> Why is it so slow, in C vs C case? (Third column) This is
worth
> investigating.
>
> Thanks,
> Samisa...
>
> Kapila Dissanayake wrote:
>
> > Hi,
> >
> > I did some testing on binary attachment transfer times,
with axis2
> > 'java' and 'C' servers with sample mtom clients. It took
very high
> > transfer times for large size attachments. Some minor
modificaions
> > were done to the data_handler.c and it was sent to the
mailing list.
> >
> >
> > (File Size) vs (Time Taken in milli seconds) to transfer
binary
> > attachments
> >
> > Environment
> > ----------
> > Processor - Intel Celeron M 1.40GHz
> > RAM - 256 MB
> > OS - Fedora Core 4.
> > Java Server - axis2 Java
> > C Server - axis2
> > mtom C and java test clients were used
> > Time Taken in milli seconds
> >
> > File Size (kb)
> >
> > Java Server
> >
> > (ms)
> >
> > Java Server
> >
> > (ms)
> >
> > C Server
> >
> > (ms)
> >
> > C Server
> >
> > (ms)
> >
> >
> >
> >
> > Java Client
> >
> > (ms)
> >
> > C Client
> >
> > (ms)
> >
> > C Client
> >
> > (ms)
> >
> > Java Client
> >
> > (ms)
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> >
> > ---------------
> > 28
> >
> >
> >
> > 445
> >
> >
> >
> > 274
> >
> >
> >
> > 2004
> >
> >
> >
> > 1009
> >
> >
> > 38
> >
> >
> >
> > 617
> >
> >
> >
> > 297
> >
> >
> >
> > 1957
> >
> >
> >
> > 842
> >
> >
> > 132
> >
> >
> >
> > 593
> >
> >
> >
> > 527
> >
> >
> >
> > 2181
> >
> >
> >
> > 726
> >
> >
> > 154
> >
> >
> >
> > 551
> >
> >
> >
> > 447
> >
> >
> >
> > 2081
> >
> >
> >
> > 677
> >
> >
> > 165
> >
> >
> >
> > 440
> >
> >
> >
> > 743
> >
> >
> >
> > 2147
> >
> >
> >
> > 676
> >
> >
> > 561
> >
> >
> >
> > 1029
> >
> >
> >
> > 418
> >
> >
> >
> > 2581
> >
> >
> >
> > 1075
> >
> >
> > 762
> >
> >
> >
> > 2137
> >
> >
> >
> > 1572
> >
> >
> >
> > 3231
> >
> >
> >
> > 2356
> >
> >
> > 1162
> >
> >
> >
> > 3172
> >
> >
> >
> > 2962
> >
> >
> >
> > 3981
> >
> >
> >
> > 3100
> >
> >
> > 1761
> >
> >
> >
> > 7672
> >
> >
> >
> > 5494
> >
> >
> >
> > 8186
> >
> >
> >
> > 3041
> >
> >
> > 2081
> >
> >
> >
> > 7768
> >
> >
> >
> > 6837
> >
> >
> >
> > 11757
> >
> >
> >
> > 3499
> >
> >
> > 4241
> >
> >
> >
> > 14465
> >
> >
> >
> > 24060
> >
> >
> >
> > 36168
> >
> >
> >
> > Could not transfer (Error Occured in Java client)
> >
> >
> >
> >
> > Thank You
> >
> > Kapila
>
>
>
------------------------------------------------------------------------
#include <stdio.h>
#include <sys/stat.h>
int read_from_file(char *file_name, char** output_stream, int
*output_stream_size);
int main(int argc, char **argv)
{
int size;
char *output;
if (argc != 2)
{
printf("usage: read_binary <file_name>\n");
return 0;
}
if (read_from_file(argv[1], &output, &size))
printf("read successful\n");
if (output)
free(output);
return 1;
}
int read_from_file(char *file_name, char** output_stream, int
*output_stream_size)
{
FILE *f = NULL;
struct stat stat_p;
int count = 0;
if (!file_name)
{
printf("error: file not given \n");
return 0;
}
f = fopen(file_name, "rb");
if (!f)
{
printf("error:cannot open file %s\n", file_name);
return ;
}
if ( -1 == stat (file_name, &stat_p))
{
printf("error:error reading stat\n");
return 0;
}
*output_stream_size = stat_p.st_size;
printf("size = %d\n", *output_stream_size);
*output_stream = malloc((*output_stream_size) * sizeof(char));
if (!(*output_stream))
{
printf("error: cannot create stream\n");
return 0;
}
count = fread(*output_stream, 1, *output_stream_size, f);
if (ferror(f) != 0)
{
printf("error:cannot read file\n");
if (*output_stream)
{
free(*output_stream);
*output_stream = NULL;
}
return 0;
}
fclose(f);
return 1;
}
