Hi Matthias,
I am working on SYSTEMML-2169 issue. I have sent a partially completed PR (
https://github.com/apache/systemml/pull/747 ). After those changes, some
test cases in NaryRBindTest, are failing and I could not understand the
reason.
Test cases are failing with following error
*Caused by: org.apache.sysml.runtime.DMLRuntimeException: Mismatched block
sizes for: 280 101 1000 101                                            *
* at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:622)*
* at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:596)*

Even after debugging the whole process I could not find a reason for this.
If you can give any suggestion that would be really helpful.

If you have any other comment regarding the PR I could modify code
according to that.

Thanks,
Chamath


On Wed, Mar 7, 2018 at 11:44 AM, Matthias Boehm <mboe...@gmail.com> wrote:

> ---------- Forwarded message ----------
> From: Matthias Boehm <mboe...@gmail.com>
> Date: Tue, Mar 6, 2018 at 10:14 PM
> Subject: Re: Extending Codegen algorithm tests for heuristics
> To: Chamath Abeysinghe <abeysinghecham...@gmail.com>
>
>
> Hi Chamath,
>
> great thanks for your contribution - I left a couple of comments but we
> should be ready to merge this in soon. If you want to get a better feeling
> for the distributed spark backend as well, I created SYSTEMML-2169, which
> aims to extend our recently added nary cbind/rbind operations to leverage
> broadcasts when applicable.
>
> Regarding the proposal, most of the backends are rather independent, but
> each backend depends on the language integration. We will help out where
> necessary. So it depends on your interests and ideas. If you're more
> interested in defining the language APIs, make this and a simple backend
> the core of your proposal. If you're more interested in the runtime
> backends, I would help and add a basic language integration in time, which
> would allow you to immediate start working on the backends.
>
> Following the GSoC guidelines it's usually better to underscope the project
> than overscope it because you want to ensure that you're able to
> successfully complete the project in the ambitious timeframe and there will
> always be unforeseen obstacles. I would recommend to define a core project
> and potential extensions you will address if time allows. For example, the
> local, multi-threaded backend can indeed be realized relatively quickly.
> However, subsequently we can add and experiment with Hogwild! (i.e.,
> unsynchronized updates) which is known to work well for sparse models,
> replication and partitioning in NUMA settings, and potentially the
> automatic selection of update strategies.
>
> Regards,
> Matthias
>
>
> On Tue, Mar 6, 2018 at 10:24 AM, Chamath Abeysinghe <
> abeysinghecham...@gmail.com> wrote:
>
> > Hi,
> > I have sent a pull request for this issue.
> > As a next step, could you suggest any new issue? or anything I have to do
> > to familiarize with Language and run time for parameter servers project.
> >
> > And regarding writing the project proposal I have few questions.
> > * In the epic there are few sub tasks, is it enough to focus on a single
> > task through out the summer? Would it have enough work load or should I
> go
> > for multiple tasks?
> > * What is the linkage between sub tasks? Do tasks like, Distributed Spark
> > Back-end or Local multi threaded back ends; need previous tasks completed
> > before starting work?
> >
> > I am glad if you could suggests some issues related to Distributed spark
> > back-end or multi threaded backend tasks.
> >
> > Thanks.
> > Regards,
> > Chamath
> >
> >
> > On Fri, Mar 2, 2018 at 6:46 AM, Matthias Boehm <mboe...@gmail.com>
> wrote:
> >
> >> Hi Chamath,
> >>
> >> in general, you're absolutely right - you can enable -stats and
> >> programmatically probe the heavy hitter statistics for certain opcodes.
> >> However, uamin and uamax stand for "unary aggregate minimum" and "unary
> >> aggregation maximum" which correspond to min(X) and max(X) on script
> level.
> >> Instead all generated fused operators are prefixed with spoof or
> sp_spoof
> >> (for distributed spark operations). The related junit assertion should
> >> already be in the existing tests, I just mentioned it for completeness.
> >>
> >> Regards,
> >> Matthias
> >>
> >> On Thu, Mar 1, 2018 at 4:30 AM, Chamath Abeysinghe <
> >> abeysinghecham...@gmail.com> wrote:
> >>
> >>> Thanks for your detailed reply.
> >>> I did some coding
> >>> <https://github.com/apache/systemml/compare/master...
> chamathabeysinghe:SYSTEMML-2159?diff=split&name=SYSTEMML-2159>
> >>> [1] for this issue SYSTEMML-2159 to extend test cases for FA & FNR . I
> got
> >>> a problem regarding success criteria's, "generating at least one fused
> >>> operator" condition, I think this means I have to look into stats of
> Heavy
> >>> hitter instructions and check if there are any fused operators. (my
> guess
> >>> is uamin and uamax are the operators what I have to look for, but I am
> not
> >>> sure about this because I don't know the meaning of these
> instructions).
> >>>
> >>> Please help me to clarify this. If my approach is correct I could send
> a
> >>> PR after fixing tests for other algorithms. Thanks.
> >>>
> >>> Regards,
> >>> Chamath
> >>>
> >>> [1] https://github.com/apache/systemml/compare/master...cham
> >>> athabeysinghe:SYSTEMML-2159?diff=split&name=SYSTEMML-2159
> >>>
> >>>
> >>> On Tue, Feb 27, 2018 at 1:54 AM, Matthias Boehm <mboe...@gmail.com>
> >>> wrote:
> >>>
> >>>> ---------- Forwarded message ----------
> >>>> From: Matthias Boehm <mboe...@gmail.com>
> >>>> Date: Mon, Feb 26, 2018 at 11:59 AM
> >>>> Subject: Re: Extending Codegen algorithm tests for heuristics
> >>>> To: Chamath Abeysinghe <abeysinghecham...@gmail.com>
> >>>>
> >>>>
> >>>> great - thanks for taking this over Chamath.
> >>>>
> >>>> In general, I would recommend to use this task to explore SystemML a
> >>>> little. For example, take one of the codegen algorithm tests from
> >>>> org.apache.sysml.test.integration.functions.codegenalg (e.g.,
> >>>> AlgorithmL2SVM) and pass different flags such as -stats, -explain,
> >>>> -explain
> >>>> recompile_hops, -explain recompile_runtime to programArgs and try to
> >>>> understand the output. If you come over specific questions, please
> just
> >>>> ask.
> >>>>
> >>>> To answer your detailed questions:
> >>>>
> >>>> 1) We recently added a code generation framework that automatically
> >>>> identifies opportunities for fused operators and subsequently
> generates
> >>>> code for these operators. A major part is the selection of fusion
> plans,
> >>>> for which we provide heuristics and a cost-based optimizer. By default
> >>>> (and
> >>>> thus also in our testsuite), we use the cost-based optimizer, but it
> >>>> would
> >>>> be good regularly test the heuristics as well.
> >>>>
> >>>> 2) You can configure the used optimizer in your SystemML-config.xml
> >>>> file as
> >>>> follows:
> >>>> <sysml.codegen.optimizer>fuse_all</sysml.codegen.optimizer>
> >>>> Valid alternatives are: fuse_all, fuse_no_redundancy, fuse_cost_based,
> >>>> and
> >>>> fuse_cost_based_v2 (default). You can provide alternative config xml
> >>>> files
> >>>> and switch them dynamically via getConfigTemplateFile.
> >>>>
> >>>> 3) Similar to the existing tests, it needs to (1) run without errors,
> >>>> (2)
> >>>> produce correct results as compared to R, and (3) generate at least
> one
> >>>> fused operator.
> >>>>
> >>>> Regards,
> >>>> Matthias
> >>>>
> >>>> On Mon, Feb 26, 2018 at 6:54 AM, Chamath Abeysinghe <
> >>>> abeysinghecham...@gmail.com> wrote:
> >>>>
> >>>> > Hi All,
> >>>> > As per the guidelines given to GSoC students, I would like to work
> on
> >>>> the
> >>>> > SYSTEMML-2159 [1] issue as a starting point. But I don't understand
> >>>> the
> >>>> > background of the issue. Can someone help me with understanding the
> >>>> context
> >>>> > of this issue?
> >>>> >
> >>>> > Few problems I got are,
> >>>> >
> >>>> > 1) What are fusion heuristics, fuse-all and fuse-no-redundancy?
> >>>> > 2) Can I pass those heuristic related configurations as args to
> >>>> execute
> >>>> > DMLScript?
> >>>> > 3) What is the success criteria for a test that use those
> heuristics?
> >>>> >
> >>>> > Thank you in advance
> >>>> >
> >>>> > Regards,
> >>>> > Chamath
> >>>> >
> >>>> > [1] https://issues.apache.org/jira/browse/SYSTEMML-2159
> >>>> >
> >>>> > --
> >>>> > Chamath Abeysinghe
> >>>> > Department of Computer Science and Engineering
> >>>> > University of Moratuwa
> >>>> >   <https://www.facebook.com/chamath.abeysinghe.3>  [image:
> >>>> > https://www.linkedin.com/in/kaushalya-gayan-batawala-bbb5927
> >>>> a?trk=hp-identity-name]
> >>>> > <https://lk.linkedin.com/in/chamathabeysinghe>
> >>>> > Mobile : +94752930548
> >>>> >
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Chamath Abeysinghe
> >>> Department of Computer Science and Engineering
> >>> University of Moratuwa
> >>>   <https://www.facebook.com/chamath.abeysinghe.3>  [image:
> >>> https://www.linkedin.com/in/kaushalya-gayan-batawala-
> bbb5927a?trk=hp-identity-name]
> >>> <https://lk.linkedin.com/in/chamathabeysinghe>
> >>> Mobile : +94752930548
> >>>
> >>
> >>
> >
> >
> > --
> > Chamath Abeysinghe
> > Department of Computer Science and Engineering
> > University of Moratuwa
> > Mobile: +94712803295 <+94%2071%20280%203295>
> >
>



-- 
Chamath Abeysinghe
Department of Computer Science and Engineering
University of Moratuwa
Mobile: +94712803295

Reply via email to