[ https://issues.apache.org/jira/browse/VELOCITY-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014900#comment-16014900 ]
James R Doyle commented on VELOCITY-880: ---------------------------------------- As suggested above, I tried: InputStream rawStream = rs.getBinaryStream(templateColumn); All tests pass except the testUnicode test - which results in mangled output org.junit.ComparisonFailure: Unicode test failed. Expected :The Euro Currency Symbol € is a two-byte UTF-8 character. Actual :The Euro Currency Symbol � is a two-byte UTF-8 character. getCharacterStream() is the safest, and it works with both VARCHAR and CLOB columns. The reason I use IOUtils to convert the Reader to an InputStream is that the base class method expects InpuStream: return buildReader(rawStream, encoding); Given what we now know about #getBinaryStream(), the scope of this bug now includes the 1.7.x branch. Again, the broken unit test fails to reveal this. I'd love to fix this problem for you, I've been using a workaround (custom Resource Loader) for a few years. Being a good citizen sharing the problem. Database hosted Velocity Templates are in production on two large clients of mine - one being a prestigious University and another a US Federal agency. I'd love to help fix this bug in the main release pool. How shall I proceed? > DataSourceResourceLoader corrupts UTF-8 encoded characters in template > ---------------------------------------------------------------------- > > Key: VELOCITY-880 > URL: https://issues.apache.org/jira/browse/VELOCITY-880 > Project: Velocity > Issue Type: Bug > Affects Versions: 2.1.x > Environment: Oracle12c and HSQLDB 2.3.4, JDK 1.8 > Reporter: James R Doyle > Attachments: velocity-880.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > A long-withstanding bug in the DataSourceResourceLoader corrupts UTF-8 > templates retrieved from the database. The Unit Test suite for this resource > loader has deficiencies that hide the bug. > The cause of the problem is this: > {code} > InputStream rawStream = rs.getAsciiStream(templateColumn); > The resolution of the problem is simply: > Reader r = rs.getCharacterStream(templateColumn); > InputStream rawStream = null; > try { > rawStream = IOUtils.toInputStream(IOUtils.toString(r), > encoding); > } catch (IOException ioe) {} > {code} > Once done, the test failure vanishes: > org.junit.ComparisonFailure: Unicode test failed. > Expected :The Euro Currency Symbol € is a two-byte UTF-8 encoded > character. > Actual :The Euro Currency Symbol ? is a two-byte UTF-8 encoded > character. > The bug was verified and the fix was tested against Oracle12c and HSQLDB > 2.3.4 using a CLOB column to store the template data. > The Unit Tests for this resource loader need attention. > Please see VELOCITY-599 ; long standing problem, which has been erroneously > marked as resolved but has been in the codebase for a long time. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@velocity.apache.org For additional commands, e-mail: dev-h...@velocity.apache.org