[jira] [Updated] (SYSTEMML-1292) Support spark codegen instructions w/ multiple RDD inputs
[ https://issues.apache.org/jira/browse/SYSTEMML-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-1292: - Summary: Support spark codegen instructions w/ multiple RDD inputs (was: Generate n-ary rdd operations) > Support spark codegen instructions w/ multiple RDD inputs > - > > Key: SYSTEMML-1292 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1292 > Project: SystemML > Issue Type: Sub-task > Components: Compiler, Runtime >Reporter: Matthias Boehm > Fix For: SystemML 1.0 > > > This task aims to support spark codegen instructions (for all templates) over > multiple RDD inputs if not all side inputs fit into the local and remote > broadcast memory budgets. In detail, this might entail either (1) generating > custom RDD operations and functions for various combinations of input RDDs, > or (2) a generalization of the related spark instructions regarding the input > RDD construction and a generic function signature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1292) Generate n-ary rdd operations
[ https://issues.apache.org/jira/browse/SYSTEMML-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-1292: - Description: This task aims to support spark codegen instructions (for all templates) over multiple RDD inputs if not all side inputs fit into the local and remote broadcast memory budgets. In detail, this might entail either (1) generating custom RDD operations and functions for various combinations of input RDDs, or (2) a generalization of the related spark instructions regarding the input RDD construction and a generic function signature. > Generate n-ary rdd operations > - > > Key: SYSTEMML-1292 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1292 > Project: SystemML > Issue Type: Sub-task > Components: Compiler, Runtime >Reporter: Matthias Boehm > Fix For: SystemML 1.0 > > > This task aims to support spark codegen instructions (for all templates) over > multiple RDD inputs if not all side inputs fit into the local and remote > broadcast memory budgets. In detail, this might entail either (1) generating > custom RDD operations and functions for various combinations of input RDDs, > or (2) a generalization of the related spark instructions regarding the input > RDD construction and a generic function signature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SYSTEMML-1782) Extend parfor block indexing analysis
[
https://issues.apache.org/jira/browse/SYSTEMML-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hu reassigned SYSTEMML-1782:
Assignee: Fei Hu
> Extend parfor block indexing analysis
> -
>
> Key: SYSTEMML-1782
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1782
> Project: SystemML
> Issue Type: Task
>Reporter: Matthias Boehm
>Assignee: Fei Hu
>
> This task aims to better support arbitrary block indexing in a parfor
> context, for expressions as this example from SYSTEMML-1774:
> {code}
> parfor (j in 1:parallel_batches, log=DEBUG) {
> # Get a mini-batch in this group
> beg = ((j-1) * batch_size) %% nrow(X_group_batch) + 1
> end = min(nrow(X_group_batch), beg + batch_size - 1)
> X_batch = X_group_batch[beg:end,]
> y_batch = y_group_batch[beg:end,]
> {code}
> Currently, we're unable to determine a constant size for X_batch and y_batch.
> However, since the parfor body is anyway recompiled on parfor entry, we can
> improve upon this as follows:
> * Leverage known constants and index ranges to perform a limit analysis for
> linear functions in indexing expressions. For example, the above expression
> for beg and end would simplify to {{(j-1) * 32 + 1}} and {{beg + batch_size
> -1}}.
> * Better handling of linear functions in right indexing HOPs - this would
> allow us to determine batch_size as the number of rows in the output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SYSTEMML-1837) Unary aggregate w/ corrections output to large physical blocks
Matthias Boehm created SYSTEMML-1837:
Summary: Unary aggregate w/ corrections output to large physical
blocks
Key: SYSTEMML-1837
URL: https://issues.apache.org/jira/browse/SYSTEMML-1837
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
Many unary aggregate operations store corrections in additional columns or
rows. For example, {{rowSums(X)}} uses a two-column output to store sums and
corrections. In CP, we drop these corrections immediately after the operations,
while in MR and Spark these corrections are dropped after final aggregation.
The issue is that the {{MatrixBlock::dropLastRowsOrColums}} does not actually
drop the correction but simply shifts all values in the right starting
positions. Hence, the physical output is actually larger than what the memory
estimates represent. This leads to unnecessary large memory consumption during
subsequent operations and in the buffer pool, which can lead to OOMs. This task
aims to fix {{MatrixBlock::dropLastRowsOrColums}}.
In a subsequent task, we could also modify all unary aggregates to never
allocate the multi-column/row output when executed in CP. However, this
requires custom code paths for the different backends.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1834) Fusion plan enumeration: short-circuit costing by upper bound
[ https://issues.apache.org/jira/browse/SYSTEMML-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Hutchison resolved SYSTEMML-1834. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Merged in [#611|https://github.com/apache/systemml/pull/611] > Fusion plan enumeration: short-circuit costing by upper bound > - > > Key: SYSTEMML-1834 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1834 > Project: SystemML > Issue Type: Improvement > Components: Compiler >Reporter: Dylan Hutchison >Assignee: Dylan Hutchison >Priority: Minor > Fix For: SystemML 1.0 > > > An intensive step during codegen is enumerating alternative ways to fuse Hops > together. The search process during this step computes the cost of many > alternative plans. We should speed up the search process by stopping the > costing process early when a cost exceeds that of the best plan found so far > (an upper bound on the best plan in the search space). > Measurements show that >60% of costed plans can be stopped early, at least > for the StratStats script. It is not clear how early the plans can be > stopped, but there is a definite benefit in search time (in seconds). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
