I successfully managed to run a program that compresses and decompresses
files using the examples in the wiki. But after looking more in depth to
the examples, I noticed I still don't know how to get access to the
internal buffer where the data is being stored. I tried pumping to an
ArraySink. However, I get data corruptions when I do that.
try
{
MeterFilter meter;
Gunzip filter;
byte* buff = new byte[4096];
ofstream of;
of.open("result.mp4", ofstream::out | ofstream::binary);
FileSource source("compress.bin", false);
ArraySink sink2(buff, 4096);
//FileSink sink("result.mp4");
source.Attach(new Redirector(filter));
filter.Attach(new Redirector(meter));
meter.Attach(new Redirector(sink2));
//meter.Attach(new Redirector(sink));
const word64 BLOCK_SIZE = 4096;
word64 remaining = FileSize(source);
word64 processed = 0;
while(remaining && !source.SourceExhausted())
{
unsigned int req = STDMIN(remaining, BLOCK_SIZE);
source.Pump(req);
filter.Flush(false);
of.write((char*)buff, req);
processed += req;
remaining -= req;
if (processed % (1024*1024*10) == 0)
cout << "Processed: " << meter.GetTotalBytes() << endl;
}
of.close();
delete[] buff;
filter.MessageEnd();
}
catch(const Exception& ex)
{
cerr << ex.what() << endl;
return 1;
}
return 0;
The compression method I'm using pipes the data directly to a FileSink, and
it works (the decompression method also works if I do that), but when I try
to get the uncompressed data in memory trough an ArraySink and then pass it
trough a file stream, it seems to fail at some point. I have to be able to
handle the processed data after pumping and processing it, and before
placing it into a file, rather than passing it directly to a file.
It seems like what I'm doing works and I get the uncompressed data, but at
some point it stops storing data or the data is simply corrupted. The
original file is 145.1 MiB long, the compressed file is 144.7 MiB, and the
resultant uncompressed file is 144.7 MiB again. Examining the data in an
hex editor shows nothing unusual in the resultant file (I can see the
headers and the binary information). Using:
cmp TEST/video.mp4 result.mp4
Reports the following:
TEST/video.mp4 result.mp4 differ: byte 4097, line 19
El martes, 2 de mayo de 2017, 18:30:33 (UTC+2), Aarón CdC escribió:
>
> I found examples on the wiki about how to use crypto++ to compress and
> decompress files using Crypto++'s Gzip algorithm. But all examples I found
> process the whole file at once. Imagine I need to compress a 1GB+ file.
> Maybe now we have computers with large amounts of memory, but this is
> completely out of the question.
>
> What I'm trying to do is to process a file using buffers to load small
> chunks of data trough a File Source. This is what I got so far:
>
> ofstream output;
>
> //Compression
> output.open("TEST/out.a", fstream::binary);
> int got,ncomp,diff = 0;
> char* ib = new char [BUFSIZE];
> char* ob = new char [BUFSIZE];
>
> for(int x = 0; x < BUFSIZE; x++)
> {
> ib[x] |= 0x0;
> ob[x] |= 0x0;
> }
>
>
>
> FileSource fsr("TEST/a.jpg", false, new Gzip(new ArraySink((byte*)
> ob, BUFSIZE)));
> while(!fsr.SourceExhausted())
> {
> got = fsr.Pump(BUFSIZE);
> fsr.Flush(false);
> cout << "Pumped " << got << " bytes" << endl;
> if(got == 0)
> break;
> output.write(ob, got);
> }
>
> output.close();
>
> //Decompression
>
> for(int x = 0; x < BUFSIZE; x++)
> {
> ib[x] |= 0x0;
> ob[x] |= 0x0;
> }
>
> cout << "Decript" << endl;
> output.open("TEST/test.jpg", fstream::binary);
>
> FileSource fir("TEST/out.a", false, new Gunzip(new ArraySink((byte
> *)ib, BUFSIZE)));
> while(!fir.SourceExhausted())
> {
> got = fir.Pump(BUFSIZE);
> fir.Flush(false);
> cout << "Pumped " << got << " bytes" << endl;
> if(got == 0)
> break;
> output.write(ib, got);
> }
>
> output.close();
> delete[] ib, ob;
> cout << "Done" << endl;
>
> I have several problems with this code. First of all, the file is not
> processed entirely. Instead, one portion is processed, and then it repeats
> trough the whole file. This is obviously not what I want to do, I need to
> process the whole file, but in small chunks. Also, SourceExhausted() doesnt
> return true once it reaches the end of the file (thus why there is a break
> in this code when there shouldn't be needed).
>
> I know there are ways to do this directly without the need of a buffer,
> but I need to pass it trough memory because I need to implement this
> somewhere else, and I need all this data to be processed first in memory,
> so I cant use a FileSink.
>
--
--
You received this message because you are subscribed to the "Crypto++ Users"
Google Group.
To unsubscribe, send an email to cryptopp-users-unsubscr...@googlegroups.com.
More information about Crypto++ and this group is available at
http://www.cryptopp.com.
---
You received this message because you are subscribed to the Google Groups
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
