[jira] [Updated] (TORQUE-364) RecordMapper very slow on many columns in table

2024-05-06 Thread Georg Kallidis (Jira)


 [ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Georg Kallidis updated TORQUE-364:
--
Fix Version/s: 6.0

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
> Fix For: 6.0
>
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)
> credits to [~refarb] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org



[jira] [Updated] (TORQUE-364) RecordMapper very slow on many columns in table

2023-04-16 Thread Max Philipp Wriedt (Jira)


 [ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Max Philipp Wriedt updated TORQUE-364:
--
Description: 
When "doSelect()" a large quantity of columns in a table the default 
RecordMappers generated by Om-Templates (processRow()) cause an  
!https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
 Problem. (technically O(rows * columns))

Specifically, constantly generating the SQL expression of all possible columns 
for every row in the result causes excessive use of StringBuilders which slow 
the mapping process to a crawl.

I currently have two ideas on how best to tackle this problem:
 # Either generate all SQL column expressions once when a (template generated) 
RecordMapper is first created (using final static String fields) thus reducing 
the cost for every row to generating all selected column SQL expressions  
once(instead of every selected column times every available column)
 # Or (in case the first approach generates unacceptably excessive number of 
fields for RecordMappers) adjust the RecordMapper API to allow a 
"prepare(Criteria, int offset)" method to be called once before processing any 
rows and implement it on generated RecordMappers to scan the Criteria and build 
two lists: One containing references to the setXXX methods of the mapper in the 
order they appear in the ResultSet (via the order in the Criteria) and a second 
list containing the corresponding column offsets. This would allow the 
processRow method to only iterated over both lists simultaneously and call the 
referenced methods with the result set and offset immediately. (Alternatively 
one list using lambdas could be used but I am currently not sure about the 
stance or impact of these lambdas in the Torque project.)

credits to [~refarb] 

  was:
When "doSelect()" a large quantity of columns in a table the default 
RecordMappers generated by Om-Templates (processRow()) cause an  
!https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
 Problem. (technically O(rows * columns))



Specifically, constantly generating the SQL expression of all possible columns 
for every row in the result causes excessive use of StringBuilders which slow 
the mapping process to a crawl.

I currently have two ideas on how best to tackle this problem:
 # Either generate all SQL column expressions once when a (template generated) 
RecordMapper is first created (using final static String fields) thus reducing 
the cost for every row to generating all selected column SQL expressions  
once(instead of every selected column times every available column)
 # Or (in case the first approach generates unacceptably excessive number of 
fields for RecordMappers) adjust the RecordMapper API to allow a 
"prepare(Criteria, int offset)" method to be called once before processing any 
rows and implement it on generated RecordMappers to scan the Criteria and build 
two lists: One containing references to the setXXX methods of the mapper in the 
order they appear in the ResultSet (via the order in the Criteria) and a second 
list containing the corresponding column offsets. This would allow the 
processRow method to only iterated over both lists simultaneously and call the 
referenced methods with the result set and offset immediately. (Alternatively 
one list using lambdas could be used but I am currently not sure about the 
stance or impact of these lambdas in the Torque project.)


> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates 

[jira] [Updated] (TORQUE-364) RecordMapper very slow on many columns in table

2023-04-15 Thread Max Philipp Wriedt (Jira)


 [ 
https://issues.apache.org/jira/browse/TORQUE-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Max Philipp Wriedt updated TORQUE-364:
--
Labels: criteria_api om performance recordmapper templates  (was: )

> RecordMapper very slow on many columns in table
> ---
>
> Key: TORQUE-364
> URL: https://issues.apache.org/jira/browse/TORQUE-364
> Project: Torque
>  Issue Type: Improvement
>  Components: Runtime, Templates
>Affects Versions: 5.1
>Reporter: Max Philipp Wriedt
>Priority: Major
>  Labels: criteria_api, om, performance, recordmapper, templates
>
> When "doSelect()" a large quantity of columns in a table the default 
> RecordMappers generated by Om-Templates (processRow()) cause an  
> !https://wikimedia.org/api/rest_v1/media/math/render/svg/4441d9689c0e6b2c47994e2f587ac5378faeefba!
>  Problem. (technically O(rows * columns))
> Specifically, constantly generating the SQL expression of all possible 
> columns for every row in the result causes excessive use of StringBuilders 
> which slow the mapping process to a crawl.
> I currently have two ideas on how best to tackle this problem:
>  # Either generate all SQL column expressions once when a (template 
> generated) RecordMapper is first created (using final static String fields) 
> thus reducing the cost for every row to generating all selected column SQL 
> expressions  once(instead of every selected column times every available 
> column)
>  # Or (in case the first approach generates unacceptably excessive number of 
> fields for RecordMappers) adjust the RecordMapper API to allow a 
> "prepare(Criteria, int offset)" method to be called once before processing 
> any rows and implement it on generated RecordMappers to scan the Criteria and 
> build two lists: One containing references to the setXXX methods of the 
> mapper in the order they appear in the ResultSet (via the order in the 
> Criteria) and a second list containing the corresponding column offsets. This 
> would allow the processRow method to only iterated over both lists 
> simultaneously and call the referenced methods with the result set and offset 
> immediately. (Alternatively one list using lambdas could be used but I am 
> currently not sure about the stance or impact of these lambdas in the Torque 
> project.)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: torque-dev-unsubscr...@db.apache.org
For additional commands, e-mail: torque-dev-h...@db.apache.org