GitHub user sandhyasun opened a pull request:
https://github.com/apache/trafodion/pull/1503
[TRAFOODION-2949] Performance changes for LOB and increasing limit for
larger varchars/â¦
â¦bobs to use with batch input.
For LOB internal descriptor files, there was a check in compiler that
prevented caching of NA Table . Removed that restriction for LOB descriptor
tables which affected evey DML into a table with a LOB column.
Fix to allow larger varchar/lob data to be input with batch input.
1. First check for rowset size is in pre codegen phase. This was
added during POC since we were getting 4 GB of rowset data all at once.
CQD MEMORY_LIMIT_ROWSET_IN_MB was added and set to 500MB. The data +
rowset size needed to fit into this space. I have changed it to be 1GB by
defaults now.
2. Then we check how big the up queuesize should be to handle this rowset
.
CQD EXE_MEMORY_FOR_UNPACK_ROWS_IN_MB is set to 100MB for this. In this
case we dynamically adjust the queue size to accommodate larger rows. Eg is we
have 100 rows of 2 MB each, we will limit upqueue size to 50 and allow 50
entries in upqueue.
I have changed the default value for this CQD to be 1GB to match (1.)
3. At runtime, the problem is that after passing these 2 checks, we try to
allocate double the size of the up queue size for the full row length .
Assume we calculate in step(2) the upqueue size is N, then we allocate
rowlength*N*2 at runtime. This clearly exceeds what we have estimated at
compiletime.
This change includes a change in compiler to estimate the memory correctly.
So with these we can handle upto 16MB of long varchar/LOB data for batch
input data a little better. But driver will still need to ensure that the
rowset buffer will fit into the 1GB limit otherwise an error will be returned
at compiletime. So for batch input for large rows , we need to limit number of
rows in the rowset.
Also included a fix for TRAFODION-909
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sandhyasun/trafodion traf_lob_global_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafodion/pull/1503.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1503
----
commit 528e9c60345234e554d32db637c82728b63c945b
Author: Sandhya Sundaresan <sandhya@...>
Date: 2018-03-29T05:56:15Z
Performance changes for LOB and increasing limit for larger varchars/bobs
to use with batch input.
----
---
