[
https://issues.apache.org/jira/browse/SYSTEMML-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149666#comment-16149666
]
Matthias Boehm commented on SYSTEMML-1782:
------------------------------------------
I would recommend to extend the parfor optimizer by this analysis. Note that
parfor optimization is invoked during runtime on entering the parfor loop.
Hence, the range and increment of {{j}} become known and can be used to perform
this range indexing analysis. Specifically, we want to determine if beg:end is
constant for all assignments of {{j}}. For linear functions, we could reuse the
parfor dependency analysis framework, while for non-linear operations like min
and max, we might need to do a limit analysis and break it into piece-wise
linear functions.
> Extend parfor block indexing analysis
> -------------------------------------
>
> Key: SYSTEMML-1782
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1782
> Project: SystemML
> Issue Type: Task
> Reporter: Matthias Boehm
> Assignee: Fei Hu
>
> This task aims to better support arbitrary block indexing in a parfor
> context, for expressions as this example from SYSTEMML-1774:
> {code}
> parfor (j in 1:parallel_batches, log=DEBUG) {
> # Get a mini-batch in this group
> beg = ((j-1) * batch_size) %% nrow(X_group_batch) + 1
> end = min(nrow(X_group_batch), beg + batch_size - 1)
> X_batch = X_group_batch[beg:end,]
> y_batch = y_group_batch[beg:end,]
> {code}
> Currently, we're unable to determine a constant size for X_batch and y_batch.
> However, since the parfor body is anyway recompiled on parfor entry, we can
> improve upon this as follows:
> * Leverage known constants and index ranges to perform a limit analysis for
> linear functions in indexing expressions. For example, the above expression
> for beg and end would simplify to {{(j-1) * 32 + 1}} and {{beg + batch_size
> -1}}.
> * Better handling of linear functions in right indexing HOPs - this would
> allow us to determine batch_size as the number of rows in the output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
