Apologies - I should have read all my e-mail before I started replying.
Ignore last message on this :>.
You are quite correct - I missed the logic for handling large buffers. I will update the CVS version this weekend.
On the second problem (detailed below), bytesEaten is a reference that should return the number of bytes eaten from the source buffer. So it isn't necessarily putting a chNull at the end of the buffer.
However, it will *only* work if the characters in the input buffer are all single byte. It will fail miserably for multi-byte characters. So your patch is absolutely the correct way to catch multi-byte characters :>.
Would your input have had multi-byte characters in it? That would explain why you got the error.
Either way - both fixes are great!
Thanks!
Cheers,
Berin
Nedelcho Stanev wrote:
Hello,
When i continue to play with this function i found another problem. in the lines 347+: t->transcodeFrom(&src[totalBytesEaten], toEat, outputBuf, MYBUFSIZE, bytesEaten, charSizes); outputBuf[iReaded] = chNull;
function transcodeFrom can read less than buffer size bytes, and putting of chNull at the end of buffer is not corect.Result of this is a corrupted outputBuf, and next sbXMLChCat function cannot copy corect block of data and at the end this cause problems with large blocks of data, they are not correct.
So, finaly modified function looks like this one :
#define MYBUFSIZE 1024
XMLCh * transcodeFromUTF8(const unsigned char * src) {
// Take a UTF-8 buffer and transcode to UTF-16
safeBuffer fullDest; fullDest.sbXMLChIn(DSIGConstants::s_unicodeStrEmpty); XMLCh outputBuf[MYBUFSIZE+4];
// Used to record byte sizes unsigned char charSizes[MYBUFSIZE+4];
// Grab a transcoder XMLTransService::Codes failReason; #if defined(XSEC_XERCES_REQUIRES_MEMMGR) XMLTranscoder* t = XMLPlatformUtils::fgTransService->makeNewTranscoderFor("UTF-8", failReason, MYBUFSIZE, XMLPlatformUtils::fgMemoryManager); #else XMLTranscoder* t = XMLPlatformUtils::fgTransService->makeNewTranscoderFor("UTF-8", failReason, MYBUFSIZE); #endif
Janitor<XMLTranscoder> j_t(t);
// Need to loop through, 2K at a time
unsigned int bytesEaten; unsigned int totalBytesEaten = 0; unsigned int bytesToEat = XMLString::stringLen((char *) src);
while (totalBytesEaten < bytesToEat) { int toEat = ((bytesToEat - totalBytesEaten) > MYBUFSIZE ? MYBUFSIZE : (bytesToEat - totalBytesEaten)); int iReaded = t->transcodeFrom(&src[totalBytesEaten], toEat, outputBuf, MYBUFSIZE, bytesEaten, charSizes); outputBuf[iReaded] = chNull; fullDest.sbXMLChCat(outputBuf); totalBytesEaten += bytesEaten; }
// Dup and output
return XMLString::replicate(fullDest.rawXMLChBuffer());
} #undef MYBUFSIZE
regards, decho
