[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608900#action_12608900
]
Kristian Waagan commented on DERBY-3732:
I don't have any details, but before you changed the Blob size in the test, I
tried working with a 48 M Blob. It worked with Java SE 6, but garbage
collection "went crazy" - i.e. a lot of time was being spent on gc and I think
I saw like 12 000 collections or something. This happened during the client
test.
I tried to debug it quickly, but besides from noticing lots of rather small
byte arrays I couldn't find anything. I was unable to trace the source of these
arras during the little time I spent on the investigation.
A few "random ramblings":
How big is your page cache?
Also, unless you're already doing this, maybe it would make sense to run the
client and the server in different JVM to better monitor the heap usage?
And what about the ant / junit things? Do they use much memory?
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip2_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608879#action_12608879
]
Kathey Marsden commented on DERBY-3732:
---
I tried switching over to reflection and running with IBM and Sun JDK 1.4.2
and get an OOM in client running with 16MB heap. JDK 1.5 runs fine. Embedded
is also ok with JDK 1.4.2. I'm investigating but if anyone knows of maybe some
different behavior in client for JDK 1.4.2 that might cause this, let me know.
1)
testBlobLength(org.apache.derbyTesting.functionTests.tests.memory.BlobMemTest)java.lang.OutOfMemoryError:
at
org.apache.derby.client.am.Cursor.get_VARCHAR_FOR_BIT_DATA(Cursor.java:647)
at org.apache.derby.client.am.Cursor.getBytes(Cursor.java:1058)
at
org.apache.derby.client.am.CallableStatement.getBytesX(CallableStatement.java:676)
at
org.apache.derby.client.am.CallableLocatorProcedures.blobGetBytes(CallableLocatorProcedures.java:447)
at
org.apache.derby.client.am.BlobLocatorInputStream.readBytes(BlobLocatorInputStream.java:176)
at
org.apache.derby.client.am.BlobLocatorInputStream.read(BlobLocatorInputStream.java:135)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:237)
at java.io.BufferedInputStream.read(BufferedInputStream.java:294)
at java.io.FilterInputStream.read(FilterInputStream.java(Compiled Code))
at
org.apache.derby.client.am.CloseFilterInputStream.read(CloseFilterInputStream.java:79)
at java.io.FilterInputStream.read(FilterInputStream.java:110)
at
org.apache.derby.client.am.CloseFilterInputStream.read(CloseFilterInputStream.java:65)
at
org.apache.derbyTesting.functionTests.tests.memory.BlobMemTest.testBlobLength(BlobMemTest.java:124)
at
org.apache.derbyTesting.functionTests.tests.memory.BlobMemTest.testBlobLength(BlobMemTest.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled
Code))
at
org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:104)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.extensions.TestSetup.run(TestSetup.java:23)
at
org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.extensions.TestSetup.run(TestSetup.java:23)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.extensions.TestSetup.run(TestSetup.java:23)
at
org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
at junit.extensions.TestSetup.run(TestSetup.java:23)
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip2_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.read
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608741#action_12608741
]
Knut Anders Hatlen commented on DERBY-3732:
---
The setting of derby.storage.pageCacheSize will only be respected if the
database is booted by that test. I see it is set to 100 when it's run as a
separate test, but not when it's run as part of a larger suite.
It would be good to document in a comment why the optimizer override is needed.
I was going to suggest that we should call skip(Long.MAX_VALUE) instead of
skip(Integer.MAX_VALUE), but then I saw that it would probably trigger a bug in
ArrayInputStream.skip() (will log a separate bug for that).
Tiny nit: testBlobLength(boolean) should be private so that it's clearer that
it's not a top-level test case.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip2_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608733#action_12608733
]
Knut Anders Hatlen commented on DERBY-3732:
---
Would it be possible to compile the test against 1.4, call the lengthless
method with reflection and exclude the testBlobLengthWithLengthlessInsert test
case if the method isn't available? Then the test cases that don't use it could
run on lower Java versions.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip2_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608707#action_12608707
]
Kristian Waagan commented on DERBY-3732:
I had a look at 'derby-3732_skip2_diff.txt' and have the following comments:
a) Instead of using your own stream (Astream), can you use one of the existing
streams for testing?
Using a stream that only returns the same value over and over doesn't
detect off-by-one errors or similar (which I introduced myself and had to debug
as part of DERBY-3735). The simplest one we have is LoopingAlphabetStream for
lowercase modern Latin.
b) When reading the data back, can we use a bigger buffer than 1000 bytes?
This would speed up the test when run with the client driver. The max length
for binary varchar is the maximum that can be transferred between the server
and the client in one go with locators.
Some tiny formatting nits for SQLBinary:
1) Mix of tabs and spaces on line "throwStreamingIOException(ioe);"
2) Trailing tab on "byte[] bytes = getBytes();"
3) Last line of method "throwStreamingIOException" doesn't use tab for indent.
thanks,
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip2_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608551#action_12608551
]
Kathey Marsden commented on DERBY-3732:
---
Mike asked:
o can you explain the need for reflection code added to BaseTestCase?
I pulled that method up from AllPackages.java so I could us it in
memory._Suite().
We need to add the test by reflection because it requires jdk 1.6 because we
test lengthless setBinaryStream.
The test will be ignored lower jdk versions.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> derby-3732_skip_diff.txt, LengthLargeLob.zip, LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608387#action_12608387
]
Kristian Waagan commented on DERBY-3732:
Regarding skip performance, I would say it's hard to tell.
The code in the JVM might be optimized and perform slightly better, even if it
is using the same approach.
However, in some cases the stream implementation might be capable of skipping
an amount of data a lot faster than reading it. One example might be streams
from files, where a skip could in theory consist of just changing a pointer.
ByteArrayInputStream also has its own implementation of skip, which doesn't use
read.
Only using read directly can result in loosing optimizations for some stream
implementations.
Also note that the contracts are different for InputStream.skip and Reader.skip.
For reference, here's an excerpt from Java SE 6 InputStream.skip JavaDoc:
"Subclasses are encouraged to provide a more efficient implementation of this
method. For instance, the implementation may depend on the ability to seek."
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt,
> LengthLargeLob.zip, LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608194#action_12608194
]
Kathey Marsden commented on DERBY-3732:
---
The javadoc for skip says:
"The skip method of this class creates a byte array and then repeatedly reads
into it until n bytes have been read or the end of the stream has been reached.
"
Is there any reason that we should expect it to be faster than reading into our
own buffer?
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608171#action_12608171
]
Kristian Waagan commented on DERBY-3732:
Most implementations are "well behaved" when it comes to skip. However, we have
no guarantee of this.
At least in some places in Derby, where skipping an exact amount of bytes (or
until EOF ) is required, a skip-loop is used. If it returns 0 and there are
more bytes missing, it is confirmed by calling read. If EOF is reached, it
returns -1. If skip returned 0 because of internal buffering, read will return
the next byte and probably refill the internal buffer.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608013#action_12608013
]
Kathey Marsden commented on DERBY-3732:
---
Sorry I misspoke. I should have said it may return 0 even if the end of file
has not been reached.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608010#action_12608010
]
Knut Anders Hatlen commented on DERBY-3732:
---
> I'm not really sure the best way to test this. Should I create a new
> LowMemTest and run it with 16M heap via the top level junit-all ant
> target? This would mean it wouldn't get run in
> suites.All. Alternately I can just add tests to existing tests in
> suites.All and make sure we have coverage of the new code, but then
> it won't test for the memory usage explicitly.
What about a combination: Create a JUnit test in a separate suite and
add the suite to suites.All. Additionally, create an ant target that
runs the suite with a 16 MB heap. Then we get both the coverage and
the possibility to test the memory usage.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608007#action_12608007
]
Knut Anders Hatlen commented on DERBY-3732:
---
> I don't know that we can use skip. The javadoc seems to indicate that skip
> might return -1 even if it hasn't reached the end of the stream.
Could you point me to the javadoc that says skip() can return -1? I see that it
can return 0, but I didn't find anything about returning negative values.
http://java.sun.com/javase/6/docs/api/java/io/InputStream.html#skip(long)
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607697#action_12607697
]
Kathey Marsden commented on DERBY-3732:
---
I'm not really sure the best way to test this. Should I create a new
LowMemTest and run it with 16M heap via the top level junit-all ant target?
This would mean it wouldn't get run in suites.All. Alternately I can just add
tests to existing tests in suites.All and make sure we have coverage of the new
code, but then it won't test for the memory usage explicitly.
We have an existing test derbyStress.java which runs in the old suite which
runs with 64MB but I want to run with less memory than that.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607402#action_12607402
]
Kathey Marsden commented on DERBY-3732:
---
Thanks Mike for looking at the preview patch.
I don't know that we can use skip. The javadoc seems to indicate that skip
might return -1 even if it hasn't reached the end of the stream.
I am trying to get my head around your statement. "But I think the generic type
code also has to worry about a datatype that is coming from a user and may be a
stream. "
What user scenario is this?
I think that probably the best thing to do is only pass through the new code if
(stream instanceof Resetable), then the non-resetable stream will be
materialized as beforek,
but again I'd like to understand in what case that would occur. We do seem to
cover the getValue() case where stream is not an instanceof
FormatIdInputStream, so we must have a test case for it.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip,
> LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606869#action_12606869
]
Kathey Marsden commented on DERBY-3732:
---
Looking at a possible quick fix approach as opposed to reworking the type code.
If I insert the lob using length, I should have a streamValueLength available.
When we call getLength() stream is set but streamValueLength is not. Below is
the trace when the stream is set. It seems to me readRecordFromArray should
be able to tell the length and pass that to setStream(). Does that sound
reasonable?
SQLBlob(SQLBinary).setStream(InputStream) line: 563
StoredPage.readRecordFromArray(Object[], int, int[], int[],
ArrayInputStream, StoredRecordHeader, RecordHandle) line: 5592
StoredPage.restoreRecordFromSlot(int, Object[], FetchDescriptor,
RecordHandle, StoredRecordHeader, boolean) line: 1497
StoredPage(BasePage).fetchFromSlot(RecordHandle, int, Object[],
FetchDescriptor, boolean) line: 459
HeapScan(GenericScanController).fetchRows(DataValueDescriptor[][],
RowLocation[], BackingStoreHashtable, long, int[]) line: 759
HeapScan.fetchNextGroup(DataValueDescriptor[][], RowLocation[]) line:
324
BulkTableScanResultSet.reloadArray() line: 327
BulkTableScanResultSet.getNextRowCore() line: 282
ProjectRestrictResultSet.getNextRowCore() line: 255
ProjectRestrictResultSet(BasicNoPutResultSetImpl).getNextRow() line:
460
EmbedResultSet40(EmbedResultSet).movePosition(int, int, String) line:
423
EmbedResultSet40(EmbedResultSet).next() line: 367
LengthLargeLob.main(String[]) line: 21
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: LengthLargeLob.zip, LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3732) SQL Length function materializes lob into memory
[
https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606676#action_12606676
]
Kristian Waagan commented on DERBY-3732:
Of course this bug should be fixed, but as Mike suggested in another Jira
issue, I think it's time to look a bit closer on the code and see if we need a
more thorough rewrite. The SQLXX classes themselves comes to mind, but we
should also verify that there isn't a mismatch between what the SQLXX classes
expect/assumes and what the store is capable of. Last, we also have a number of
LOB specific classes on top.
For instance, would it be wiser to more clearly separate code for LOBs from the
smaller data types in the data type hierarchy?
I don't know the history of the code, but it might be problematic if the
existing code was written with small data types in mind.
There seems to be at least two separate issues:
a) Materialization
b) Stream handling
As a sub point under b, I'm also wondering why we don't encode stream length in
the store stream and how hard it would be to fix this. One problem is that the
space reserved for this purpose seems too limited.
> SQL Length function materializes lob into memory
>
>
> Key: DERBY-3732
> URL: https://issues.apache.org/jira/browse/DERBY-3732
> Project: Derby
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>Reporter: Kathey Marsden
>Priority: Minor
> Attachments: LengthLargeLob.zip
>
>
> Currently the SQL length function materializes the entire lob into memory. In
> SQLBinary.getLength() we have
> public final int getLength() throws StandardException
> {
> if (stream != null) {
> if (streamValueLength != -1)
> return streamValueLength;
> }
> return (getBytes() == null) ? 0 : getBytes().length;
> }
> Which actually is doubly bad because we call getBytes twice and materialize
> it twice.
> It would be good to read the length from the stream if available and
> otherwise stream the value to get the length, rather than materializing it
> into memory.
> To reproduce, run the attached repro.
> java -Xmx16M LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
> at
> org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
> at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
> at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
> at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
> at
> org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
> at
> org.apache.derby.exe.acf81e0010x011axa317x5db8x003d9dc81.e1(Unknown
> Source)
> at
> org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
> at
> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
> at
> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
> ... 2 more
> [
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
