[jira] [Assigned] (SYSTEMML-1657) Python package should force usage of associated JAR
[ https://issues.apache.org/jira/browse/SYSTEMML-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner reassigned SYSTEMML-1657: --- Assignee: Glenn Weidner > Python package should force usage of associated JAR > --- > > Key: SYSTEMML-1657 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1657 > Project: SystemML > Issue Type: Bug > Components: APIs >Reporter: Mike Dusenberry >Assignee: Glenn Weidner > > Currently, our Python package checks if a SystemML JAR is already present on > the classpath, and if not, attempts to load the JAR that is shipped with the > package. We should reverse this, and instead always try to use the > associated JAR first, and if it is not available (such as during a local pip > dev install without the JAR packaged), then look into the current classpath > to see if the JAR is available. The reasoning is that in a hosted > environment that has a version of SystemML preinstalled (i.e. the JAR is > available on the classpath and the Python package is installed, with the idea > of the underlying system being ready for either Python or Scala usage), if > the user attempts to update the Python package via pip, the old JAR will > still be on the classpath (since it is preinstalled), and thus the older JAR > (i.e. 0.14) will be used by the newer Python package (i.e 1.0 dev), leading > to issues. For extra clarity, we still want the Python package to fall back > on a JAR on the classpath if a JAR is not part of the package, such as during > local dev installs. We would just prefer that the Python package use the > packaged JAR first. > Here is the associated code: > https://github.com/apache/incubator-systemml/blob/master/src/main/python/systemml/classloader.py#L64. > Essentially, the try/catch block needs to be reversed, assuming we can for > the JVM classloader to load the packaged JAR with priority to any existing > SystemML JARs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1701) Fix the need to add force to -gpu always
Nakul Jindal created SYSTEMML-1701: -- Summary: Fix the need to add force to -gpu always Key: SYSTEMML-1701 URL: https://issues.apache.org/jira/browse/SYSTEMML-1701 Project: SystemML Issue Type: Sub-task Components: Runtime Affects Versions: SystemML 0.14 Reporter: Nakul Jindal Assignee: Nakul Jindal Fix For: SystemML 1.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1694) Add snapshot version number to docs header
[ https://issues.apache.org/jira/browse/SYSTEMML-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1694. > Add snapshot version number to docs header > -- > > Key: SYSTEMML-1694 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1694 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Gus Jenkins > Fix For: SystemML 1.0 > > > Currently the latest snapshot documentation > (http://apache.github.io/systemml/) has "Latest" in the header. "Latest" can > be a little confusing since it's hard to tell whether this is the latest > release version (currently 0.14.0-incubating) or the latest snapshot version > (1.0.0-SNAPSHOT). > We should probably change "Latest" to either something like: > Latest (1.0.0-SNAPSHOT) > or > 1.0.0-SNAPSHOT -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1625) Add (Unit) Tests for GPU functions
[ https://issues.apache.org/jira/browse/SYSTEMML-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nakul Jindal closed SYSTEMML-1625. -- Resolution: Fixed > Add (Unit) Tests for GPU functions > -- > > Key: SYSTEMML-1625 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1625 > Project: SystemML > Issue Type: Sub-task > Components: Test >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1694) Add snapshot version number to docs header
[ https://issues.apache.org/jira/browse/SYSTEMML-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1694. -- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed by [PR545|https://github.com/apache/systemml/pull/545]. > Add snapshot version number to docs header > -- > > Key: SYSTEMML-1694 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1694 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Gus Jenkins > Fix For: SystemML 1.0 > > > Currently the latest snapshot documentation > (http://apache.github.io/systemml/) has "Latest" in the header. "Latest" can > be a little confusing since it's hard to tell whether this is the latest > release version (currently 0.14.0-incubating) or the latest snapshot version > (1.0.0-SNAPSHOT). > We should probably change "Latest" to either something like: > Latest (1.0.0-SNAPSHOT) > or > 1.0.0-SNAPSHOT -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1679:
--
Description:
We should add a new {{threshold}} utility function (in {{nn/util.dml}}) that
accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
indicator matrix {{out}} with values in \{0, 1\} depending on whether or not
the values in {{X}} are above {{thresh}}. We could use this, for example, for
determining the predicted class in a binary classification problem given the
output of a sigmoid layer.
We should also add a test case in {{nn/test/test.dml}}.
was:We should add a new {{threshold}} utility function (in {{nn/util.dml}})
that accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
indicator matrix {{out}} with values in \{0, 1\} depending on whether or not
the values in {{X}} are above {{thresh}}. We could use this, for example, for
determining the predicted class in a binary classification problem given the
output of a sigmoid layer.
> Add a new threshold utility function
>
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add a new {{threshold}} utility function (in {{nn/util.dml}}) that
> accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
> indicator matrix {{out}} with values in \{0, 1\} depending on whether or not
> the values in {{X}} are above {{thresh}}. We could use this, for example,
> for determining the predicted class in a binary classification problem given
> the output of a sigmoid layer.
> We should also add a test case in {{nn/test/test.dml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1698) Deep Learning Built-In Functions not appear in menu of DML Lang Ref
[ https://issues.apache.org/jira/browse/SYSTEMML-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson resolved SYSTEMML-1698. -- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed by [PR544|https://github.com/apache/systemml/pull/544]. > Deep Learning Built-In Functions not appear in menu of DML Lang Ref > --- > > Key: SYSTEMML-1698 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1698 > Project: SystemML > Issue Type: Bug > Components: Documentation >Reporter: Deron Eriksson >Assignee: Fei Hu > Fix For: SystemML 1.0 > > > The "Deep Learning Built-In Functions" section does not appear in the menu at > the top of the DML Language Reference > (http://apache.github.io/systemml/dml-language-reference.html). > It should be added after "Data Pre-Processing Built-In Functions" and before > "Other Built-In Functions". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1698) Deep Learning Built-In Functions not appear in menu of DML Lang Ref
[ https://issues.apache.org/jira/browse/SYSTEMML-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deron Eriksson closed SYSTEMML-1698. > Deep Learning Built-In Functions not appear in menu of DML Lang Ref > --- > > Key: SYSTEMML-1698 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1698 > Project: SystemML > Issue Type: Bug > Components: Documentation >Reporter: Deron Eriksson >Assignee: Fei Hu > Fix For: SystemML 1.0 > > > The "Deep Learning Built-In Functions" section does not appear in the menu at > the top of the DML Language Reference > (http://apache.github.io/systemml/dml-language-reference.html). > It should be added after "Data Pre-Processing Built-In Functions" and before > "Other Built-In Functions". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1700) Basic develeoper documentation
Matthias Boehm created SYSTEMML-1700: Summary: Basic develeoper documentation Key: SYSTEMML-1700 URL: https://issues.apache.org/jira/browse/SYSTEMML-1700 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1699) Improve IPA propagation of consistent nnz per input
Matthias Boehm created SYSTEMML-1699: Summary: Improve IPA propagation of consistent nnz per input Key: SYSTEMML-1699 URL: https://issues.apache.org/jira/browse/SYSTEMML-1699 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1698) Deep Learning Built-In Functions not appear in menu of DML Lang Ref
Deron Eriksson created SYSTEMML-1698: Summary: Deep Learning Built-In Functions not appear in menu of DML Lang Ref Key: SYSTEMML-1698 URL: https://issues.apache.org/jira/browse/SYSTEMML-1698 Project: SystemML Issue Type: Bug Components: Documentation Reporter: Deron Eriksson Assignee: Fei Hu The "Deep Learning Built-In Functions" section does not appear in the menu at the top of the DML Language Reference (http://apache.github.io/systemml/dml-language-reference.html). It should be added after "Data Pre-Processing Built-In Functions" and before "Other Built-In Functions". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1690) New FunctionCallCatolog for auxiliary function call information
[ https://issues.apache.org/jira/browse/SYSTEMML-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm resolved SYSTEMML-1690. -- Resolution: Done Assignee: Matthias Boehm Fix Version/s: SystemML 1.0 > New FunctionCallCatolog for auxiliary function call information > --- > > Key: SYSTEMML-1690 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1690 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1690) New FunctionCallCatolog for auxiliary function call information
[ https://issues.apache.org/jira/browse/SYSTEMML-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-1690. > New FunctionCallCatolog for auxiliary function call information > --- > > Key: SYSTEMML-1690 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1690 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1689) Refactoring into individual IPA passes
[ https://issues.apache.org/jira/browse/SYSTEMML-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm resolved SYSTEMML-1689. -- Resolution: Done Assignee: Matthias Boehm Fix Version/s: SystemML 1.0 > Refactoring into individual IPA passes > -- > > Key: SYSTEMML-1689 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1689 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1689) Refactoring into individual IPA passes
[ https://issues.apache.org/jira/browse/SYSTEMML-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-1689. > Refactoring into individual IPA passes > -- > > Key: SYSTEMML-1689 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1689 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1674) Add a new 2D depthwise convolution layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050989#comment-16050989 ] Mike Dusenberry commented on SYSTEMML-1674: --- [PR 542 | https://github.com/apache/systemml/pull/542] submitted. > Add a new 2D depthwise convolution layer > > > Key: SYSTEMML-1674 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1674 > Project: SystemML > Issue Type: Sub-task >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1674) Add a new 2D depthwise convolution layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Dusenberry updated SYSTEMML-1674: -- Summary: Add a new 2D depthwise convolution layer (was: Add a new depthwise convolution layer) > Add a new 2D depthwise convolution layer > > > Key: SYSTEMML-1674 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1674 > Project: SystemML > Issue Type: Sub-task >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1678) Add new 1D & 2D top_k utility functions
[
https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1678:
--
Description:
We should add new {{top_k}} and {{top_k2d}} utility functions (in
{{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and
{{indices}} with the top {{k}} values (i.e. probabilities) and associated
indices (i.e. classes) along a certain dimension. This will be modeled after
the [{{top_k}} function in TensorFlow |
https://www.tensorflow.org/api_docs/python/tf/nn/top_k] For the 1D case,
{{top_k}} will operate on the columns dimension. A typical use case is that in
which {{X}} is the output of a {{softmax}} layer (so each row contains a set of
normalized class probabilities), and {{values}} and {{indices}} will contain
rows with the top {{k}} probabilities and class indices as described above.
For the 2D case, {{top_k}} will operate on the channels dimension. A typical
use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so
each channel contains a set of normalized class probabilities), and {{values}}
and {{indices}} will contain the top {{k}} probabilities and indices along the
channel axis. This scenario would be common in an image segmentation problem,
in which every pixel of the output image will have a set of class probabilities
along the channel axis.
Having these {{top-k}} functions will allow us to extract either predict a
single class for each item, or the top {{k}} classes, and therefore may be more
useful that a {{predict_class}} function.
Although we will use {{values}} and {{indices}} as the names of the returned
matrices within the functions, in practice, one is likely to name the results
{{probs}} and {{classes}} in the calling environment.
was:
We should add new {{top_k}} and {{top_k2d}} utility functions (in
{{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and
{{indices}} with the top {{k}} values (i.e. probabilities) and associated
indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
will operate on the columns dimension. A typical use case is that in which
{{X}} is the output of a {{softmax}} layer (so each row contains a set of
normalized class probabilities), and {{values}} and {{indices}} will contain
rows with the top {{k}} probabilities and class indices as described above.
For the 2D case, {{top_k}} will operate on the channels dimension. A typical
use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so
each channel contains a set of normalized class probabilities), and {{values}}
and {{indices}} will contain the top {{k}} probabilities and indices along the
channel axis. This scenario would be common in an image segmentation problem,
in which every pixel of the output image will have a set of class probabilities
along the channel axis.
Having these {{top-k}} functions will allow us to extract either predict a
single class for each item, or the top {{k}} classes, and therefore may be more
useful that a {{predict_class}} function.
Although we will use {{values}} and {{indices}} as the names of the returned
matrices within the functions, in practice, one is likely to name the results
{{probs}} and {{classes}} in the calling environment.
> Add new 1D & 2D top_k utility functions
> ---
>
> Key: SYSTEMML-1678
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add new {{top_k}} and {{top_k2d}} utility functions (in
> {{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}}
> and {{indices}} with the top {{k}} values (i.e. probabilities) and associated
> indices (i.e. classes) along a certain dimension. This will be modeled after
> the [{{top_k}} function in TensorFlow |
> https://www.tensorflow.org/api_docs/python/tf/nn/top_k] For the 1D case,
> {{top_k}} will operate on the columns dimension. A typical use case is that
> in which {{X}} is the output of a {{softmax}} layer (so each row contains a
> set of normalized class probabilities), and {{values}} and {{indices}} will
> contain rows with the top {{k}} probabilities and class indices as described
> above. For the 2D case, {{top_k}} will operate on the channels dimension. A
> typical use case here is that in which {{X}} is the output of a {{softmax2d}}
> layer (so each channel contains a set of normalized class probabilities), and
> {{values}} and {{indices}} will contain the top {{k}} probabilities and
> indices along the channel axis. This scenario would be common in an image
> segmentation problem, in which every pixel of the output image will have a
> set of class
[jira] [Updated] (SYSTEMML-1678) Add new 1D & 2D top_k utility functions
[
https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1678:
--
Description:
We should add new {{top_k}} and {{top_k2d}} utility functions (in
{{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and
{{indices}} with the top {{k}} values (i.e. probabilities) and associated
indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
will operate on the columns dimension. A typical use case is that in which
{{X}} is the output of a {{softmax}} layer (so each row contains a set of
normalized class probabilities), and {{values}} and {{indices}} will contain
rows with the top {{k}} probabilities and class indices as described above.
For the 2D case, {{top_k}} will operate on the channels dimension. A typical
use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so
each channel contains a set of normalized class probabilities), and {{values}}
and {{indices}} will contain the top {{k}} probabilities and indices along the
channel axis. This scenario would be common in an image segmentation problem,
in which every pixel of the output image will have a set of class probabilities
along the channel axis.
Having these {{top-k}} functions will allow us to extract either predict a
single class for each item, or the top {{k}} classes, and therefore may be more
useful that a {{predict_class}} function.
Although we will use {{values}} and {{indices}} as the names of the returned
matrices within the functions, in practice, one is likely to name the results
{{probs}} and {{classes}} in the calling environment.
was:
We should add new {{top_k}} and {{top_k2d}} utility functions (in
{{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and
{{indices}} with the top {{k}} values (i.e. probabilities) and associated
indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
will operate on the columns dimension. A typical use case is that in which
{{X}} is the output of a {{softmax}} layer (so each row contains a set of
normalized class probabilities), and {{values}} and {{indices}} will contain
rows with the top {{k}} probabilities and class indices as described above.
For the 2D case, {{top_k}} will operate on the channels dimension. A typical
use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so
each channel contains a set of normalized class probabilities), and {{values}}
and {{indices}} will contain the top {{k}} probabilities and indices along the
channel axis. This scenario would be common in an image segmentation problem,
in which every pixel of the output image will have a set of class probabilities
along the channel axis.
Having these {{top-k}} functions will allow us to extract either predict a
single class for each item, or the top {{k}} classes, and therefore may be more
useful that a {{predict_class}} function.
> Add new 1D & 2D top_k utility functions
> ---
>
> Key: SYSTEMML-1678
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add new {{top_k}} and {{top_k2d}} utility functions (in
> {{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}}
> and {{indices}} with the top {{k}} values (i.e. probabilities) and associated
> indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
> will operate on the columns dimension. A typical use case is that in which
> {{X}} is the output of a {{softmax}} layer (so each row contains a set of
> normalized class probabilities), and {{values}} and {{indices}} will contain
> rows with the top {{k}} probabilities and class indices as described above.
> For the 2D case, {{top_k}} will operate on the channels dimension. A typical
> use case here is that in which {{X}} is the output of a {{softmax2d}} layer
> (so each channel contains a set of normalized class probabilities), and
> {{values}} and {{indices}} will contain the top {{k}} probabilities and
> indices along the channel axis. This scenario would be common in an image
> segmentation problem, in which every pixel of the output image will have a
> set of class probabilities along the channel axis.
> Having these {{top-k}} functions will allow us to extract either predict a
> single class for each item, or the top {{k}} classes, and therefore may be
> more useful that a {{predict_class}} function.
> Although we will use {{values}} and {{indices}} as the names of the returned
> matrices within the functions, in practice, one is likely to name the results
> {{probs}} and {{classes}} in the calling environment.
--
This
[jira] [Updated] (SYSTEMML-1678) Add new 1D & 2D top_k utility functions
[
https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1678:
--
Description:
We should add new {{top_k}} and {{top_k2d}} utility functions (in
{{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and
{{indices}} with the top {{k}} values (i.e. probabilities) and associated
indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
will operate on the columns dimension. A typical use case is that in which
{{X}} is the output of a {{softmax}} layer (so each row contains a set of
normalized class probabilities), and {{values}} and {{indices}} will contain
rows with the top {{k}} probabilities and class indices as described above.
For the 2D case, {{top_k}} will operate on the channels dimension. A typical
use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so
each channel contains a set of normalized class probabilities), and {{values}}
and {{indices}} will contain the top {{k}} probabilities and indices along the
channel axis. This scenario would be common in an image segmentation problem,
in which every pixel of the output image will have a set of class probabilities
along the channel axis.
Having these {{top-k}} functions will allow us to extract either predict a
single class for each item, or the top {{k}} classes, and therefore may be more
useful that a {{predict_class}} function.
was:We should add new {{predict_class}} and {{predict_class2d}} utility
functions (in {{nn/util.dml}}) that accept a matrix {{X}} and return a matrix
{{out}} with the predicted classes based on the max probability. For the 1D
case, {{X}} is the output of a {{softmax}} layer, and thus each row will
contain a set of normalized class probabilities
> Add new 1D & 2D top_k utility functions
> ---
>
> Key: SYSTEMML-1678
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add new {{top_k}} and {{top_k2d}} utility functions (in
> {{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}}
> and {{indices}} with the top {{k}} values (i.e. probabilities) and associated
> indices (i.e. classes) along a certain dimension. For the 1D case, {{top_k}}
> will operate on the columns dimension. A typical use case is that in which
> {{X}} is the output of a {{softmax}} layer (so each row contains a set of
> normalized class probabilities), and {{values}} and {{indices}} will contain
> rows with the top {{k}} probabilities and class indices as described above.
> For the 2D case, {{top_k}} will operate on the channels dimension. A typical
> use case here is that in which {{X}} is the output of a {{softmax2d}} layer
> (so each channel contains a set of normalized class probabilities), and
> {{values}} and {{indices}} will contain the top {{k}} probabilities and
> indices along the channel axis. This scenario would be common in an image
> segmentation problem, in which every pixel of the output image will have a
> set of class probabilities along the channel axis.
> Having these {{top-k}} functions will allow us to extract either predict a
> single class for each item, or the top {{k}} classes, and therefore may be
> more useful that a {{predict_class}} function.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1678) Add new 1D & 2D top_k utility functions
[
https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1678:
--
Summary: Add new 1D & 2D top_k utility functions (was: Add new 1D & 2D
predict class utility functions)
> Add new 1D & 2D top_k utility functions
> ---
>
> Key: SYSTEMML-1678
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add new {{predict_class}} and {{predict_class2d}} utility functions
> (in {{nn/util.dml}}) that accept a matrix {{X}} and return a matrix {{out}}
> with the predicted classes based on the max probability. For the 1D case,
> {{X}} is the output of a {{softmax}} layer, and thus each row will contain a
> set of normalized class probabilities
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1678) Add new 1D & 2D predict class utility functions
[
https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1678:
--
Description: We should add new {{predict_class}} and {{predict_class2d}}
utility functions (in {{nn/util.dml}}) that accept a matrix {{X}} and return a
matrix {{out}} with the predicted classes based on the max probability. For
the 1D case, {{X}} is the output of a {{softmax}} layer, and thus each row will
contain a set of normalized class probabilities
> Add new 1D & 2D predict class utility functions
> ---
>
> Key: SYSTEMML-1678
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add new {{predict_class}} and {{predict_class2d}} utility functions
> (in {{nn/util.dml}}) that accept a matrix {{X}} and return a matrix {{out}}
> with the predicted classes based on the max probability. For the 1D case,
> {{X}} is the output of a {{softmax}} layer, and thus each row will contain a
> set of normalized class probabilities
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1679:
--
Description: We should add a new {{threshold}} utility function (in
{{nn/util.dml}}) that accepts a matrix {{X}} and a threshold parameter
{{thresh}} and returns an indicator matrix {{out}} with values in \{0, 1\}
depending on whether or not the values in {{X}} are above {{thresh}}. We could
use this, for example, for determining the predicted class in a binary
classification problem given the output of a sigmoid layer. (was: We should
add a new {{threshold}} utility function (in {{nn/util.dml}}) that accepts a
matrix {{X}} and a threshold parameter {{thresh}} and returns an indicator
matrix {{out}} with values in {{{0, 1}}} depending on whether or not the values
in {{X}} are above {{thresh}}. We could use this, for example, for determining
the predicted class in a binary classification problem given the output of a
sigmoid layer.)
> Add a new threshold utility function
>
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add a new {{threshold}} utility function (in {{nn/util.dml}}) that
> accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
> indicator matrix {{out}} with values in \{0, 1\} depending on whether or not
> the values in {{X}} are above {{thresh}}. We could use this, for example,
> for determining the predicted class in a binary classification problem given
> the output of a sigmoid layer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1679:
--
Summary: Add a new threshold utility function (was: Add a new threshold
layer)
> Add a new threshold utility function
>
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add a new {{threshold}} utility function that accepts a matrix
> {{X}} and a threshold parameter {{thresh}} and returns an indicator matrix
> {{out}} with values in {{{0, 1}}} depending on whether or not the values in
> {{X}} are above {{thresh}}. We could use this, for example, for determining
> the predicted class in a binary classification problem given the output of a
> sigmoid layer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1679:
--
Description: We should add a new {{threshold}} utility function (in
{{nn/util.dml}}) that accepts a matrix {{X}} and a threshold parameter
{{thresh}} and returns an indicator matrix {{out}} with values in {{{0, 1}}}
depending on whether or not the values in {{X}} are above {{thresh}}. We could
use this, for example, for determining the predicted class in a binary
classification problem given the output of a sigmoid layer. (was: We should
add a new {{threshold}} utility function that accepts a matrix {{X}} and a
threshold parameter {{thresh}} and returns an indicator matrix {{out}} with
values in {{{0, 1}}} depending on whether or not the values in {{X}} are above
{{thresh}}. We could use this, for example, for determining the predicted
class in a binary classification problem given the output of a sigmoid layer.)
> Add a new threshold utility function
>
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add a new {{threshold}} utility function (in {{nn/util.dml}}) that
> accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
> indicator matrix {{out}} with values in {{{0, 1}}} depending on whether or
> not the values in {{X}} are above {{thresh}}. We could use this, for
> example, for determining the predicted class in a binary classification
> problem given the output of a sigmoid layer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold layer
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Dusenberry updated SYSTEMML-1679:
--
Description: We should add a new {{threshold}} utility function that
accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
indicator matrix {{out}} with values in {{{0, 1}}} depending on whether or not
the values in {{X}} are above {{thresh}}. We could use this, for example, for
determining the predicted class in a binary classification problem given the
output of a sigmoid layer.
> Add a new threshold layer
> -
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>
> We should add a new {{threshold}} utility function that accepts a matrix
> {{X}} and a threshold parameter {{thresh}} and returns an indicator matrix
> {{out}} with values in {{{0, 1}}} depending on whether or not the values in
> {{X}} are above {{thresh}}. We could use this, for example, for determining
> the predicted class in a binary classification problem given the output of a
> sigmoid layer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Dusenberry updated SYSTEMML-1679: -- Summary: Add a new threshold layer (was: Add a new threshold utility function) > Add a new threshold layer > - > > Key: SYSTEMML-1679 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1679 > Project: SystemML > Issue Type: Sub-task >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server
[ https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050871#comment-16050871 ] Niketan Pansare commented on SYSTEMML-1695: --- This is a good start. Once you are ready with your draft, you may want to send it over the mailing list for feedback from other committers :) > Create design/investigation document for parameter server > - > > Key: SYSTEMML-1695 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1695 > Project: SystemML > Issue Type: Sub-task >Reporter: Niketan Pansare > > Here are relevant links: > - https://issues.apache.org/jira/browse/SPARK-4590 > - https://issues.apache.org/jira/browse/SPARK-6932 > - > https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit > This document need not cover how to integrate a parameter server in SystemML > (which comes later), but should discuss following questions: > 1. Should we integrate existing parameter server implementation or build one > from scratch ? What are the pros and cons of each approaches ? For example: > https://github.com/rjagerman/glint > 2. If we plan to integrate existing parameter server implementation, what is > the interface ? (for exampe: update(matrix, new matrix) or update(matrix, > delta)). > 3. Can parameter server help in ML algorithms ? Some experiments might be > helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1159) Enable Remote Hyperparameter Tuning
[
https://issues.apache.org/jira/browse/SYSTEMML-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050817#comment-16050817
]
Janardhan commented on SYSTEMML-1159:
-
Hi [~dusenberrymw],
I think your idea of global sharing of whole dataset by all the models ( in
parallel ), can be implemented by the blackboard system. Please once navigate
to the brainstorming on parameter servers at systemml-1695
> Enable Remote Hyperparameter Tuning
> ---
>
> Key: SYSTEMML-1159
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1159
> Project: SystemML
> Issue Type: Improvement
>Affects Versions: SystemML 1.0
>Reporter: Mike Dusenberry
>Priority: Blocker
>
> Training a parameterized machine learning model (such as a large neural net
> in deep learning) requires learning a set of ideal model parameters from the
> data, as well as determining appropriate hyperparameters (or "settings") for
> the training process itself. In the latter case, the hyperparameters (i.e.
> learning rate, regularization strength, dropout percentage, model
> architecture, etc.) can not be learned from the data, and instead are
> determined via a search across a space for each hyperparameter. For large
> numbers of hyperparameters (such as in deep learning models), the current
> literature points to performing staged, randomized grid searches over the
> space to produce distributions of performance, narrowing the space after each
> search \[1]. Thus, for efficient hyperparameter optimization, it is
> desirable to train several models in parallel, with each model trained over
> the full dataset. For deep learning models, a mini-batch training approach
> is currently state-of-the-art, and thus separate models with different
> hyperparameters could, conceivably, be easily trained on each of the nodes in
> a cluster.
> In order to allow for the training of deep learning models, SystemML needs to
> determine a solution to enable this scenario with the Spark backend.
> Specifically, if the user has a {{train}} function that takes a set of
> hyperparameters and trains a model with a mini-batch approach (and thus is
> only making use of single-node instructions within the function), the user
> should be able to wrap this function with, for example, a remote {{parfor}}
> construct that samples hyperparameters and calls the {{train}} function on
> each machine in parallel.
> To be clear, each model would need access to the entire dataset, and each
> model would be trained independently.
> \[1]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server
[
https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050465#comment-16050465
]
Janardhan commented on SYSTEMML-1695:
-
A starting point..
{color:#14892c}https://docs.google.com/document/d/1AOW53numMJSF_msGvo1lekpyv7_3VF51i6xAjNCEC9I/edit?usp=sharing{color}
> Create design/investigation document for parameter server
> -
>
> Key: SYSTEMML-1695
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1695
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Niketan Pansare
>
> Here are relevant links:
> - https://issues.apache.org/jira/browse/SPARK-4590
> - https://issues.apache.org/jira/browse/SPARK-6932
> -
> https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit
> This document need not cover how to integrate a parameter server in SystemML
> (which comes later), but should discuss following questions:
> 1. Should we integrate existing parameter server implementation or build one
> from scratch ? What are the pros and cons of each approaches ? For example:
> https://github.com/rjagerman/glint
> 2. If we plan to integrate existing parameter server implementation, what is
> the interface ? (for exampe: update(matrix, new matrix) or update(matrix,
> delta)).
> 3. Can parameter server help in ML algorithms ? Some experiments might be
> helpful.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
