[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-05-04 Thread Max Philipp Wriedt (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843515#comment-17843515
 ] 

Max Philipp Wriedt commented on TORQUE-364:
---

The integrated code is correct as I can see for now.

As the MappingStrategy is regenerated everytime a new row is processed, this 
fix is sadly not doing anything for the performance.

However its working and we can use it for a future performance fix e.g. by 
making it reusable by saving it with the criteria or somewhere appropriate.

 

[~refarb] [~mwriedt]

 

 

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-23 Thread Georg Kallidis (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840117#comment-17840117
 ] 

Georg Kallidis commented on TORQUE-364:
---

I integrated the snippet, please check the code and message, thanks!

MappingStrategy is enabled ny default. This could be changed by an enable flag 
in options.properties in torque-generate (useMappingStrategy) and is by default 
enabled and a sorting flag (as well enabled by default). It is used in the 
template recordMapperBase.vm. Sorting might be some performance issue for table 
with  a lot columns. 

I tried to keep the code as much as it was, just inlining the new strategy 
keeping the logic and still using the HashSet. Nevertheless each call of 
processRow requires a new MappingStrategy object instance.

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-20 Thread Max Philipp Wriedt (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839298#comment-17839298
 ] 

Max Philipp Wriedt commented on TORQUE-364:
---

To point 3. we finally got a slightly nicer working prototype to share;
{code:java}
### recordMapperBase.vm
---+
// at the top import the new runtime MappingStrategy class
import org.apache.torque.om.mapper.MappingStrategy;
---
---+
// Should this be cached per RecordMapper (Thread safety/Multi query safety?)
private MappingStrategy<${dbObjectClassName}> strategy;

// A sample prepareStrategy implementation, still missing the new HashSet 
optimization!
public MappingStrategy<${dbObjectClassName}> prepareStrategy(Criteria criteria, 
int offset)
{
MappingStrategy<${dbObjectClassName}> strategy = new 
MappingStrategy<${dbObjectClassName}>();
if (criteria == null)
{
return strategy;
}

int nextOuterOffset = offset + 1;
List selectColumns = criteria.getSelectColumns();
List columnsWithoutOffset = selectColumns.subList(
offset,
selectColumns.size());

for (Column column : columnsWithoutOffset)
{
// Create a final local copy for this iteration to be passed into the 
lambdas
final int nextOffset = nextOuterOffset;
String columnExpression = column.getSqlExpression();
#set ( $else = "" )
#foreach ($columnElement in $torqueGen.getChildren("column"))
#set ( $setter = $columnElement.getAttribute("setter") )
#set ( $getter = $columnElement.getAttribute("getter") )
#set ( $peerColumnName = $columnElement.getAttribute("peerColumnName") )
#set ( $fieldType = $columnElement.getAttribute("fieldType") )
${else}if (${peerColumnName}_EXPRESSION.equals(columnExpression))
{
strategy.addColumn(nextOffset, (res, inst) -> 
inst.${setter}(this.${getter}(res, nextOffset)));
}
#set ( $else = "else ")
#end
nextOuterOffset += 1;
}
strategy.finish($torqueGen.getChildren("column").size());
return strategy;
}
---
---+
// in processRow()
// this could theoretically replace BOTh if cases (criteria == null AND != 
null)
// However criteria == null might still be slightly faster than this!

// Wherever the MappingStrategy is ultimately stored should probably handle 
invalidation
// Storing it in Criteria for example would allow invalidating whenever a 
select columns is added/modified
//  However, for testing purposes we just placed it directly in processRow 
to show viability.
if (this.strategy == null) {
this.strategy = prepareStrategy(criteria, offset);
}
this.strategy.execute(resultSet, $field);
---


### MappingStrategy.java

package org.apache.torque.om.mapper;

import org.apache.commons.lang3.tuple.Pair;
import org.apache.commons.lang3.function.FailableBiConsumer;
import org.apache.torque.TorqueException;

import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;

public class MappingStrategy {
private final List>> strategy;
private boolean allSet;

public MappingStrategy()
{
this.strategy = new ArrayList<>();
this.allSet = false;
}

public void addColumn(int offset, FailableBiConsumer setter)
{
this.strategy.add(Pair.of(offset, setter));
}

public void finish(int num_fields)
{
// The list should already be in the correct order because Criteria 
loops through the columns
// in the same order in which they are added to the SQL statement but 
just in case something weird
// is being done this gets us closer to the desired contract of 
ResultSet of looping over monotonically
// increasing indices of columns only.
this.strategy.sort(Comparator.comparing(Pair::getLeft));
this.allSet = this.strategy.size() == num_fields;
}

public boolean isEmpty()
{
return this.strategy.isEmpty();
}

public boolean isAllSet() {
return this.allSet;
}

public void execute(ResultSet result, T instance) throws TorqueException
{
for (Pair> 
strategy : this.strategy)
{
strategy.getRight().accept(result, instance);
}
}
}
{code}
This code adds a single MappingStrategy class which holds a list of lambdas 
which know how to best query a ResultSet to construct a T from it as well as 
the necessary code for a RecordMapper to utilize this MappingStrategy to speed 
up object construction significantly!
However, the actual strategy object is obviously tied to a given 
Criterias/Queries selected columns and thus can't be kept by the 
RecordMapperXXX objects themselves. (This would lead to exceptions with multple 
threads doing queries and would require extensive cache invalidation checks for 
all processRow invokations to create a new strategy even in 

[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-11 Thread Georg Kallidis (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836066#comment-17836066
 ] 

Georg Kallidis commented on TORQUE-364:
---

Regarding point 1: 

Of course it should mean BaseXXXRecordMapper . But nevertheless the criteria's 
select columns size - offset might be equal to XXX.Peer.numColumns only by 
chance, then it would fail (but might just require an additional criteria check 
e.g. criteria's fromelements is empty, but still not sure, if this is 
fail-proof).

Might be better to skip this or to allow for arbitrary customization by a user 
(who knows, what s/he does), but then the interface has to changed ..

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-10 Thread Max Philipp Wriedt (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835897#comment-17835897
 ] 

Max Philipp Wriedt commented on TORQUE-364:
---

Regarding point 1:
Our solution with the {{criteria.getSelectColumns().size()}} check only works 
if the BaseRecordMapper is used *and* never used by any other custom record 
mappers to "chain load". i.e. If I were to implement a FooAndBarMapper which 
selects 2 of 5 columns from a table Foo and 3 of 3 columns from a table Bar and 
then just call into the FooRecordMapper and BarRecordMapper with appropriate 
offsets the check would break and I would get a corrupted Foo object with 
garbage data or an exception for storing/reading invalid types from the 
ResultSet.

Regarding point 2: The following code should do the thing inside 
{{org/apache/torque/templates/om/templates/recordmapper/base/recordMapperBase.vm}}
{code:java}
## Cached SQL expressions to speed up looking for columns selected by a given 
Criteria
#foreach ($columnElement in $torqueGen.getChildren("column"))
#set ( $peerColumnName = $columnElement.getAttribute("peerColumnName") )
private static final String ${peerColumnName}_EXPRESSION = 
${basePeerClassName}.${peerColumnName}.getSqlExpression();
#end
{code}
As well as
{code:java}
String columnExpression = column.getSqlExpression();
// snip...
${else}if (${peerColumnName}_EXPRESSION.equals(columnExpression))
{code}
In the for each loop over columns

We are currently investigating on point 3 and will provide further information 
soon.

As for point 4, improvements are always nice to see.

Regards [~mwriedt] and [~refarb] (<- whose account seems to be missing?)

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-10 Thread Georg Kallidis (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835748#comment-17835748
 ] 

Georg Kallidis commented on TORQUE-364:
---

Thanks! I think there might be multiple improvements possible here:
 * The workaround you mentioned might be already correct, if narrowing down to 
only select queries (I think processRow is used in selects only, updates might 
be different) to shortcut and use the simpler (cheap) mapping. Seems alreay 
fine to be included in an updated BaseRecordMapper..
 * Prebuilding the sqlExpressions in static variables might be done quite 
easily in Torque templates and the replacing part of the conditions in the 
else-branch in processRow
 * It might be of good help, if you could provide an example implementation of 
preprocess(criteria, offset). Integration could be done either e.g. by an 
additional argument or a new interface and from there on further ..
 * The loop in the else branch could improved (it was designed with the if-else 
conditional to allow a little bit of improved evaluation IMO) also to skip the 
already checked condition by using e.g. a lookup hashset (keys might be the 
column name or the full sql expression  for the column). This way indeed a good 
part of the redundant condition evaulations is removed (it should be evaluated 
only once not columns times condition)

If I do understand this issue correctly, I might already continue by changing 
the code considering the last and the first one. For the other improvement 
options I'll wait for other's opinions and more information ..

 

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-08 Thread Max Philipp Wriedt (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834961#comment-17834961
 ] 

Max Philipp Wriedt commented on TORQUE-364:
---

I'm currently trying to work myself into this issue again, as this was quite a 
while ago.
At that time I was upgrading an old Torque-based project from version 3 to 
version 5.

The issue manifested everytime we tried to get every column instead of defining 
specific columns to show.
Our project has pretty large number of columns (over 200 at some times). 
With debugging turned on, we observed a large number of StringBuilder calls to 
create the select statement.

The BaseXXXRecordMapper function {{processRow()}} seems to be always called 
with {{{}criteria != null{}}}, even when doing a simple {{doSelect()}}
This results in mapping every column instead of directly using the getters.

Our workaround was to extend the if-else in {{{}processRow(){}}}:
{{if (criteria == null || criteria.getSelectColumns().size() - offset == 
BaseXXXPeer.numColumns)}}

We also added static Strings for every Expression, but I don't remember 
exactly, if this improved speed or something.

{{private static final String IDExpression;}}
{{static {}}
{{IDExpression = BaseXXXPeer.ID.getSqlExpression();}}
}

I will take a more detailed look into this with ~refarb in the near future. 

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Commented] (TORQUE-364) RecordMapper very slow on many columns in table

2024-04-08 Thread Georg Kallidis (Jira)


[ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834837#comment-17834837
 ] 

Georg Kallidis commented on TORQUE-364:
---

Thanks for reporting this issue. To check, if this could be included in the 
upcoming release and to tackle it down a little bit more, what about it is 
exactly and what to do, some more information would be helpful:

Indeed mappers are applied for each single row, there is some overhead here, 
but I could not find any usage of StringBuilder used in any RecordMapper 
classes. Do yo mean applying some SQL expression e.g. in (always assuming 
package {{org.apache.torque) in }}{{{}ColumnImpl.ColumnImpl(.. String 
sqlExpression){}}}? This happens of course already in building the SQL query 
(e.g {{sql.SqlBuilder.buildQuery(Criteria)) }}in doSelect() methods. The query 
is indeed build with StringBuilder in 
{{sql.Query.toStringBuilder(StringBuilder)}}

More generally this may be about some caching mechanism, e.g. applying some 
kind of save immutable context.

Could you give a more exact example, where this happens most obviously in the 
Torque template or runtime code? 

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org