[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863992#comment-15863992 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 Thanks @tillrohrmann! > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Soila Kavulya >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862934#comment-15862934 ] ASF GitHub Bot commented on FLINK-1979: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1985 > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Soila Kavulya >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862858#comment-15862858 ] ASF GitHub Bot commented on FLINK-1979: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/1985 The PR looks good to me. Thanks a lot for your contribution @skavulya. I will merge this PR once the rebased code passes Travis :-) > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862400#comment-15862400 ] ASF GitHub Bot commented on FLINK-1979: --- Github user mtunique commented on the issue: https://github.com/apache/flink/pull/1985 Could this PR be merged ? Maybe this PR blocks to design GLM([FLINK-2013](https://issues.apache.org/jira/browse/FLINK-2013)). > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440354#comment-15440354 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 hi @chiwanpark sorry, I hadn't checked this PR for a while. I merged the latest master. Do you have any preference on an alternative name for RegularizationPenalty? Would Regularizer work better? > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345779#comment-15345779 ] ASF GitHub Bot commented on FLINK-1979: --- Github user chiwanpark commented on the issue: https://github.com/apache/flink/pull/1985 Hi @skavulya, sorry for late response. I've checked the updated PR and looks good to me. I wonder whether the name `RegularizationPenalty` is proper because the class calculates lots of values. But it is trivial thing. If there is no objection in few days, I'll merge this. Thanks! > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321354#comment-15321354 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 @chiwanpark The PR is ready. Let me know if I need to do anything else. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315212#comment-15315212 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 @chiwanpark Decoupling the gradient descent step is complicated for L1 regularization because we are using the proximal gradient method that applies soft thresholding after executing the gradient descent step. I left the regularization penalty as-is. I am thinking of adding an additional method that adds the regularization penalty to gradient without the gradient descent step but I will do it in the L-BFGS PR instead. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312435#comment-15312435 ] ASF GitHub Bot commented on FLINK-1979: --- Github user chiwanpark commented on the issue: https://github.com/apache/flink/pull/1985 Okay, please ping me when the PR is updated. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311626#comment-15311626 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 While working on L-BFGS, I realized that I need to remove the gradient descent step from RegularizationPenalty. I'll update the PR soon. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311314#comment-15311314 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the issue: https://github.com/apache/flink/pull/1985 Thanks @chiwanpark. I made the changes you recommended. Please let me know if it looks ok. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309058#comment-15309058 ] ASF GitHub Bot commented on FLINK-1979: --- Github user chiwanpark commented on the pull request: https://github.com/apache/flink/pull/1985 Hi @skavulya, I just quickly reviewed your updated PR and left few comments. They are not critical things but It would be better to fix them. Other things are very good. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309056#comment-15309056
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user chiwanpark commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r65291933
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/optimization/RegularizationPenaltyTest.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.DenseVector
+import org.apache.flink.test.util.FlinkTestBase
+import org.scalatest.{FlatSpec, Matchers}
+
+
+class RegularizationPenaltyTest extends FlatSpec with Matchers with
FlinkTestBase {
--- End diff --
Same as above, we don't need to extend `FlinkTestBase`.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309054#comment-15309054
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user chiwanpark commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r65291911
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/optimization/LossFunctionTest.scala
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.common.{LabeledVector, WeightVector}
+import org.apache.flink.ml.math.DenseVector
+import org.scalatest.{Matchers, FlatSpec}
+import org.apache.flink.test.util.FlinkTestBase
+
+
+class LossFunctionTest extends FlatSpec with Matchers with FlinkTestBase {
--- End diff --
We don't need to extend `FlinkTestBase` because `LossFunctionTest` do not
use Flink cluster.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309044#comment-15309044
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user chiwanpark commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r65291552
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/RegularizationPenalty.scala
---
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.{Vector, BLAS}
+import org.apache.flink.ml.math.Breeze._
+import breeze.linalg.{norm => BreezeNorm}
+
+/** Represents a type of regularization penalty
+ *
+ * Regularization penalties are used to restrict the optimization problem
to solutions with
+ * certain desirable characteristics, such as sparsity for the L1
penalty, or penalizing large
+ * weights for the L2 penalty.
+ *
+ * The regularization term, `R(w)` is added to the objective function,
`f(w) = L(w) + lambda*R(w)`
+ * where lambda is the regularization parameter used to tune the amount
of regularization applied.
+ */
+trait RegularizationPenalty extends Serializable {
+
+ /** Calculates the new weights based on the gradient and regularization
penalty
+*
+* @param weightVector The weights to be updated
+* @param gradient The gradient used to update the weights
+* @param regularizationConstant The regularization parameter to be
applied
+* @param learningRate The effective step size for this iteration
+* @return Updated weights
+*/
+ def takeStep(
+ weightVector: Vector,
+ gradient: Vector,
+ regularizationConstant: Double,
+ learningRate: Double)
+: Vector
+
+ /** Adds regularization to the loss value
+*
+* @param oldLoss The loss to be updated
+* @param weightVector The gradient used to update the loss
+* @param regularizationConstant The regularization parameter to be
applied
+* @return Updated loss
+*/
+ def regLoss(oldLoss: Double, weightVector: Vector,
regularizationConstant: Double): Double
+
+}
+
+
+/** `L_2` regularization penalty.
+ *
+ * The regularization function is the square of the L2 norm
`1/2*||w||_2^2`
+ * with `w` being the weight vector. The function penalizes large weights,
+ * favoring solutions with more small weights rather than few large ones.
+ */
+object L2Regularization extends RegularizationPenalty {
+
+ /** Calculates the new weights based on the gradient and L2
regularization penalty
+*
+* The updated weight is `w - learningRate *(gradient + lambda * w)`
where
+* `w` is the weight vector, and `lambda` is the regularization
parameter.
+*
+* @param weightVector The weights to be updated
+* @param gradient The gradient according to which we will update the
weights
+* @param regularizationConstant The regularization parameter to be
applied
+* @param learningRate The effective step size for this iteration
+* @return Updated weights
+*/
+ override def takeStep(
+ weightVector: Vector,
+ gradient: Vector,
+ regularizationConstant: Double,
+ learningRate: Double)
+: Vector = {
+// add the gradient of the L2 regularization
+BLAS.axpy(regularizationConstant, weightVector, gradient)
+
+// update the weights according to the learning rate
+BLAS.axpy(-learningRate, gradient, weightVector)
+
+weightVector
+ }
+
+ /** Adds regularization to the loss value
+*
+* The updated loss is `oldLoss + lambda * 1/2*||w||_2^2` where
+* `w` is the weight vector, and `lambda` is the regularization
parameter
+*
+* @param oldLoss The loss to
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309043#comment-15309043
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user chiwanpark commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r65291461
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/PartialLossFunction.scala
---
@@ -47,21 +47,106 @@ object SquaredLoss extends PartialLossFunction {
/** Calculates the loss depending on the label and the prediction
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
*/
override def loss(prediction: Double, label: Double): Double = {
0.5 * (prediction - label) * (prediction - label)
}
/** Calculates the derivative of the [[PartialLossFunction]]
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The derivative of the loss function
*/
override def derivative(prediction: Double, label: Double): Double = {
(prediction - label)
}
}
+
+/** Logistic loss function which can be used with the
[[GenericLossFunction]]
+ *
+ *
+ * The [[LogisticLoss]] function implements `log(1 +
-exp(prediction*label))`
+ * for binary classification with label in {-1, 1}
+ */
+object LogisticLoss extends PartialLossFunction {
+
+ /** Calculates the loss depending on the label and the prediction
+*
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
+*/
+ override def loss(prediction: Double, label: Double): Double = {
+val z = prediction * label
+
+// based on implementation in scikit-learn
+// approximately equal and saves the computation of the log
+if (z > 18) {
+ return math.exp(-z)
+}
+else if (z < -18) {
+ return -z
+}
+
+math.log(1 + math.exp(-z))
+ }
+
+ /** Calculates the derivative of the loss function with respect to the
prediction
+*
+* @param prediction The predicted value
+* @param label The true value
+* @return The derivative of the loss function
+*/
+ override def derivative(prediction: Double, label: Double): Double = {
+val z = prediction * label
+
+// based on implementation in scikit-learn
+// approximately equal and saves the computation of the log
+if (z > 18) {
+ return -label * math.exp(-z)
+}
+else if (z < -18) {
+ return -label
+}
+
+-label/(math.exp(z) + 1)
+ }
--- End diff --
As I said above, following is better:
```scala
if (z > 18) {
-label * math.exp(-z)
} else if (z < -18) {
-label
} else {
-label / (math.exp(z) + 1)
}
```
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309038#comment-15309038
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user chiwanpark commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r65291353
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/PartialLossFunction.scala
---
@@ -47,21 +47,106 @@ object SquaredLoss extends PartialLossFunction {
/** Calculates the loss depending on the label and the prediction
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
*/
override def loss(prediction: Double, label: Double): Double = {
0.5 * (prediction - label) * (prediction - label)
}
/** Calculates the derivative of the [[PartialLossFunction]]
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The derivative of the loss function
*/
override def derivative(prediction: Double, label: Double): Double = {
(prediction - label)
}
}
+
+/** Logistic loss function which can be used with the
[[GenericLossFunction]]
+ *
+ *
+ * The [[LogisticLoss]] function implements `log(1 +
-exp(prediction*label))`
+ * for binary classification with label in {-1, 1}
+ */
+object LogisticLoss extends PartialLossFunction {
+
+ /** Calculates the loss depending on the label and the prediction
+*
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
+*/
+ override def loss(prediction: Double, label: Double): Double = {
+val z = prediction * label
+
+// based on implementation in scikit-learn
+// approximately equal and saves the computation of the log
+if (z > 18) {
+ return math.exp(-z)
+}
+else if (z < -18) {
+ return -z
+}
+
+math.log(1 + math.exp(-z))
--- End diff --
Using `return` is not recommended in Scala. Could you change this like
following?
```scala
if (z > 18) {
math.exp(-z)
} else if (z < -18) {
-z
} else {
math.log(1 + math.exp(-z))
}
```
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308199#comment-15308199 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the pull request: https://github.com/apache/flink/pull/1985 @tillrohrmann @thvasilo I made the changes you recommended. Please let me know if they look ok. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288559#comment-15288559
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63657433
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/optimization/RegularizationPenaltyITSuite.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.DenseVector
+import org.apache.flink.test.util.FlinkTestBase
+import org.scalatest.{FlatSpec, Matchers}
+
+
+class RegularizationPenaltyITSuite extends FlatSpec with Matchers with
FlinkTestBase {
--- End diff --
You're right @thvasilo. This would be not an integration test because it
tests the regularization component without firing up a Flink cluster and
executing an actual job. So we could remove the `ITSuite` from the name.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287019#comment-15287019
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user skavulya commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63562423
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/optimization/RegularizationPenaltyITSuite.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.DenseVector
+import org.apache.flink.test.util.FlinkTestBase
+import org.scalatest.{FlatSpec, Matchers}
+
+
+class RegularizationPenaltyITSuite extends FlatSpec with Matchers with
FlinkTestBase {
--- End diff --
Thanks @tillrohrmann @thvasilo for all your comments. I'll make the
changes for passing the regression penalty as a parameter, fix the formatting
issue and update the unit tests
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287013#comment-15287013
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user skavulya commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63561773
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -272,7 +272,7 @@ abstract class GradientDescent extends IterativeSolver {
* The regularization function is `1/2 ||w||_2^2` with `w` being the
weight vector.
*/
class GradientDescentL2 extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
I can add it to this PR since it is only affecting SGD.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286595#comment-15286595
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63519957
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/optimization/RegularizationPenaltyITSuite.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.DenseVector
+import org.apache.flink.test.util.FlinkTestBase
+import org.scalatest.{FlatSpec, Matchers}
+
+
+class RegularizationPenaltyITSuite extends FlatSpec with Matchers with
FlinkTestBase {
--- End diff --
@tillrohrmann Do tests like these that only test components without an
`ExecutionEnvironment` being created get named differently (i.e. no ITSuite in
the class name)
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286591#comment-15286591
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63519390
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/RegularizationPenalty.scala
---
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.optimization
+
+import org.apache.flink.ml.math.{Vector, BLAS}
+import org.apache.flink.ml.math.Breeze._
+import breeze.linalg.{norm => BreezeNorm}
+
+/** Represents a type of regularization penalty
+ *
+ * Regularization penalties are used to restrict the optimization problem
to solutions with
+ * certain desirable characteristics, such as sparsity for the L1
penalty, or penalizing large
+ * weights for the L2 penalty.
+ *
+ * The regularization term, `R(w)` is added to the objective function,
`f(w) = L(w) + lambda*R(w)`
+ * where lambda is the regularization parameter used to tune the amount
of regularization applied.
+ */
+trait RegularizationPenalty extends Serializable {
+
+ /** Calculates the new weights based on the gradient and regularization
penalty
+*
+* @param weightVector The weights to be updated
+* @param gradient The gradient used to update the weights
+* @param regularizationConstant The regularization parameter to be
applied
+* @param learningRate The effective step size for this iteration
+* @return Updated weights
+*/
+ def takeStep(
+ weightVector: Vector,
+ gradient: Vector,
+ regularizationConstant: Double,
+ learningRate: Double)
+: Vector
+
+ /** Adds regularization to the loss value
+*
+* @param oldLoss The loss to be updated
+* @param weightVector The gradient used to update the loss
+* @param regularizationConstant The regularization parameter to be
applied
+* @return Updated loss
+*/
+ def regLoss(oldLoss: Double, weightVector: Vector,
regularizationConstant: Double): Double
+
+}
+
+
+/** `L_2` regularization penalty.
+ *
+ * The regularization function is the square of the L2 norm
`1/2*||w||_2^2`
+ * with `w` being the weight vector. The function penalizes large weights,
+ * favoring solutions with more small weights rather than few large ones.
+ */
+object L2Regularization extends RegularizationPenalty {
+
+ /** Calculates the new weights based on the gradient and L2
regularization penalty
+*
+* The updated weight is `w - learningRate *(gradient + lambda * w)`
where
+* `w` is the weight vector, and `lambda` is the regularization
parameter.
+*
+* @param weightVector The weights to be updated
+* @param gradient The gradient according to which we will update the
weights
+* @param regularizationConstant The regularization parameter to be
applied
+* @param learningRate The effective step size for this iteration
+* @return Updated weights
+*/
+ override def takeStep(
+ weightVector: Vector,
+ gradient: Vector,
+ regularizationConstant: Double,
+ learningRate: Double)
+: Vector = {
+// add the gradient of the L2 regularization
+BLAS.axpy(regularizationConstant, weightVector, gradient)
+
+// update the weights according to the learning rate
+BLAS.axpy(-learningRate, gradient, weightVector)
+
+weightVector
+ }
+
+ /** Adds regularization to the loss value
+*
+* The updated loss is `l + lambda * 1/2*||w||_2^2` where `l` is the
old loss,
--- End diff --
I would recommend spelling out `loss` here, as in some fonts it's hard tell
the letter
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286577#comment-15286577
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63517587
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -272,7 +272,7 @@ abstract class GradientDescent extends IterativeSolver {
* The regularization function is `1/2 ||w||_2^2` with `w` being the
weight vector.
*/
class GradientDescentL2 extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
I've mentioned this in the previous PR but adding here for completeness:
I'm in favor of adding the regularization penalty as a parameter for the
optimizer.
However that would involve changes that perhaps beyond the scope of this
PR,
currently with only SGD available we don't have to worry about the
applicability of L1/L2 regularization, but should add a note for when L-BFGS
get implemented.
Depending on how much work @skavulya it would be to make the change here,
we can choose to have a separate PR for that or include it here.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286578#comment-15286578
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63517610
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -272,7 +272,7 @@ abstract class GradientDescent extends IterativeSolver {
* The regularization function is `1/2 ||w||_2^2` with `w` being the
weight vector.
*/
class GradientDescentL2 extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
I've mentioned this in the previous PR but adding here for completeness:
I'm in favor of adding the regularization penalty as a parameter for the
optimizer.
However that would involve changes that perhaps beyond the scope of this
PR,
currently with only SGD available we don't have to worry about the
applicability of L1/L2 regularization, but should add a note for when L-BFGS
get implemented.
Depending on how much work @skavulya it would be to make the change here,
we can choose to have a separate PR for that or include it here.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286498#comment-15286498 ] ASF GitHub Bot commented on FLINK-1979: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/1985#issuecomment-219702326 Thanks for your contribution @skavulya. The changes look really good :-) I especially like your thorough and clear documentation of the different loss functions. Great work. I had some minor comments. Once fixed, the PR should be good to be merged :-) > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286480#comment-15286480
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63510204
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/PartialLossFunction.scala
---
@@ -47,21 +47,106 @@ object SquaredLoss extends PartialLossFunction {
/** Calculates the loss depending on the label and the prediction
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
*/
override def loss(prediction: Double, label: Double): Double = {
0.5 * (prediction - label) * (prediction - label)
}
/** Calculates the derivative of the [[PartialLossFunction]]
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The derivative of the loss function
*/
override def derivative(prediction: Double, label: Double): Double = {
(prediction - label)
}
}
+
+/** Logistic loss function which can be used with the
[[GenericLossFunction]]
+ *
+ *
+ * The [[LogisticLoss]] function implements `log(1 +
exp(prediction*label))`
--- End diff --
A minus is missing in the exponent of e.
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286471#comment-15286471
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509887
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/PartialLossFunction.scala
---
@@ -47,21 +47,106 @@ object SquaredLoss extends PartialLossFunction {
/** Calculates the loss depending on the label and the prediction
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The loss
*/
override def loss(prediction: Double, label: Double): Double = {
0.5 * (prediction - label) * (prediction - label)
}
/** Calculates the derivative of the [[PartialLossFunction]]
*
-* @param prediction
-* @param label
-* @return
+* @param prediction The predicted value
+* @param label The true value
+* @return The derivative of the loss function
--- End diff --
Good code completion, thanks :-)
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286469#comment-15286469 ] ASF GitHub Bot commented on FLINK-1979: --- Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/1985#discussion_r63509731 --- Diff: flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/LossFunction.scala --- @@ -23,8 +23,8 @@ import org.apache.flink.ml.math.BLAS /** Abstract class that implements some of the functionality for common loss functions * - * A loss function determines the loss term $L(w) of the objective function $f(w) = L(w) + - * \lambda R(w)$ for prediction tasks, the other being regularization, $R(w)$. + * A loss function determines the loss term `L(w)` of the objective function `f(w) = L(w) + + * lambda*R(w)` for prediction tasks, the other being regularization, `R(w)`. --- End diff -- Good catch :-) > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286466#comment-15286466
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509544
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -364,8 +350,7 @@ class SimpleGradientDescent extends GradientDescent {
learningRate: Double)
: Vector = {
// Update the weight vector
-BLAS.axpy(-learningRate, gradient, weightVector)
-weightVector
+NoRegularization.takeStep(weightVector, gradient,
regularizationConstant,learningRate)
--- End diff --
whitespace
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286464#comment-15286464
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509514
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -321,19 +318,8 @@ class GradientDescentL1 extends GradientDescent {
regularizationConstant: Double,
learningRate: Double)
: Vector = {
-// Update weight vector with gradient. L1 regularization has no
gradient, the proximal operator
-// does the job.
-BLAS.axpy(-learningRate, gradient, weightVector)
-
-// Apply proximal operator (soft thresholding)
-val shrinkageVal = regularizationConstant * learningRate
-var i = 0
-while (i < weightVector.size) {
- val wi = weightVector(i)
- weightVector(i) = scala.math.signum(wi) *
-scala.math.max(0.0, scala.math.abs(wi) - shrinkageVal)
- i += 1
-}
+
+L1Regularization.takeStep(weightVector, gradient,
regularizationConstant,learningRate)
--- End diff --
whitespace missing
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286465#comment-15286465
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509537
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -348,7 +334,7 @@ object GradientDescentL1 {
* No regularization is applied.
*/
class SimpleGradientDescent extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
TODO
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286460#comment-15286460
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509381
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -272,7 +272,7 @@ abstract class GradientDescent extends IterativeSolver {
* The regularization function is `1/2 ||w||_2^2` with `w` being the
weight vector.
*/
class GradientDescentL2 extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
Is this TODO still valid? If so, can we resolve it?
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286463#comment-15286463
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509482
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -306,7 +303,7 @@ object GradientDescentL2 {
* The regularization function is `||w||_1` with `w` being the weight
vector.
*/
class GradientDescentL1 extends GradientDescent {
-
+ //TODO(skavulya): Pass regularization penalty as a parameter
--- End diff --
TODOs are always good to be resolved :-)
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286461#comment-15286461
]
ASF GitHub Bot commented on FLINK-1979:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1985#discussion_r63509413
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/optimization/GradientDescent.scala
---
@@ -287,11 +287,8 @@ class GradientDescentL2 extends GradientDescent {
regularizationConstant: Double,
learningRate: Double)
: Vector = {
-// add the gradient of the L2 regularization
-BLAS.axpy(regularizationConstant, weightVector, gradient)
-// update the weights according to the learning rate
-BLAS.axpy(-learningRate, gradient, weightVector)
+L2Regularization.takeStep(weightVector, gradient,
regularizationConstant,learningRate)
--- End diff --
whitespace missing between comma and `learningRate`
> Implement Loss Functions
>
>
> Key: FLINK-1979
> URL: https://issues.apache.org/jira/browse/FLINK-1979
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
>Reporter: Johannes Günther
>Assignee: Johannes Günther
>Priority: Minor
> Labels: ML
>
> For convex optimization problems, optimizer methods like SGD rely on a
> pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279807#comment-15279807 ] ASF GitHub Bot commented on FLINK-1979: --- Github user thvasilo commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-218399619 Hello @skavulya thank you for your contribution, I'm looking forward to the PR! Re. the regularization penalty, we it changed to make user choice easier, e.g. preventing the use of an L1 penalty when L-BFGS is used. Personally I would prefer to go back to the original design and have it as a settable parameter, and apply some sanity checking within the optimizer to avoid cases like the one mentioned above. I would suggest you open the PR and we can discuss it there. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279692#comment-15279692 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-218380988 @tillrohrmann I updated the loss functions and regularization penalties based on your branch. I was not sure whether to update the gradient descent algorithm to pass the regularization penalty as a parameter in this branch so I kept the GradientDescent API as-is. Let me know if you would like me to change the GradientDescent API to GradientDescent().setRegularizationPenalty(L1Regularization) and I will update my branch, squash the commits, and create a PR. https://github.com/apache/flink/compare/master...skavulya:loss-functions > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240781#comment-15240781 ] ASF GitHub Bot commented on FLINK-1979: --- Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-209818726 Sure go ahead @skavulya. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239918#comment-15239918 ] ASF GitHub Bot commented on FLINK-1979: --- Github user skavulya commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-209618076 I need this pull request for work I am doing with L-BFGS. I would like to take it over and make the recommended changes since there has not been any activity on this for a while. > Implement Loss Functions > > > Key: FLINK-1979 > URL: https://issues.apache.org/jira/browse/FLINK-1979 > Project: Flink > Issue Type: Improvement > Components: Machine Learning Library >Reporter: Johannes Günther >Assignee: Johannes Günther >Priority: Minor > Labels: ML > > For convex optimization problems, optimizer methods like SGD rely on a > pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593383#comment-14593383 ] ASF GitHub Bot commented on FLINK-1979: --- Github user thvasilo commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-113504786 Hello, this PR needs to be rebase since so that it's mergeable with the latest master. I think it's fair to say that we can close it in a week if no more activity occurs. Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565253#comment-14565253 ] ASF GitHub Bot commented on FLINK-1979: --- Github user aalexandrov commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-106907665 This needs a PR before it is merged. @jojo19893, @mguldner please cherry pick your changes such that you have only one or two commits and force push in this branch as suggested by @tillrohrmann in the JIRA. Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550452#comment-14550452 ] Till Rohrmann commented on FLINK-1979: -- The easiest way would probably be to simply cherry pick your commits onto the current master branch and to update the pull request. Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537855#comment-14537855 ] ASF GitHub Bot commented on FLINK-1979: --- Github user thvasilo commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-100884364 Thank you Johaness. The optimization code has been merged to the master now, so could you rebase your branch to the latest master so we can look at the changes in an isolated way? You can take a look at the [How to contibute](http://flink.apache.org/how-to-contribute.html#contributing-code--documentation) guide on how to do this. The merges you have currently make it hard to review the code. Also, please make sure all your classes have docstrings, you can take the docstring for SquaredLoss as an example (i.e. one sentence is usually enough). Documentation is always welcome of course, so if you want to add some more details to the loss functions section of the ML documentation (docs/libs/ml/optimization.md) feel free to do so in this PR. Let me know if you run into any problems. Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537342#comment-14537342 ] ASF GitHub Bot commented on FLINK-1979: --- Github user jojo19893 commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-100696661 Okay I Changed the implementation of the logistic loss! Thanks for the advice ! Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536365#comment-14536365 ] Johannes Günther commented on FLINK-1979: - Okay I will adjust the code so that it matches the implementation that was suggested. I will probably do it tommorrow or on Monday Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534138#comment-14534138 ] ASF GitHub Bot commented on FLINK-1979: --- Github user thvasilo commented on the pull request: https://github.com/apache/flink/pull/656#issuecomment-100159626 Thank you for the contribution Johaness. For the logistic loss could you change the implementation to match the approximate version that sklearn (and other libraries) use? You can see it [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/sgd_fast.pyx#L202), this way we may save the computation of the log. @tillrohrmann Any objections to using this version? Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534219#comment-14534219 ] Till Rohrmann commented on FLINK-1979: -- Sounds like a good idea to save some computations. Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532401#comment-14532401 ] Johannes Günther commented on FLINK-1979: - Okay I changed the Pull request to the original Flink Repository ! Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532398#comment-14532398 ] ASF GitHub Bot commented on FLINK-1979: --- GitHub user jojo19893 opened a pull request: https://github.com/apache/flink/pull/656 Lossfunctions We added Logistic Loss Functions and Hinge Loss to the Optimazation Framework. See for the implemented Functions: https://github.com/JohnLangford/vowpal_wabbit/wiki/Loss-functions Jira Issue: https://issues.apache.org/jira/browse/FLINK-1979 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jojo19893/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/656.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #656 commit 4431e1d2ed0ebfd230ae997bcbf412c965108034 Author: Theodore Vasiloudis t...@sics.se Date: 2015-04-21T08:59:34Z [FLINK-1807] [ml] Adds optimization framework and SGD solver. Added Stochastic Gradient Descent initial version and some tests. Added L1, L2 regularization. Added tests for regularization, fixed parameter setting. commit e51c63583514d51f727816944df01a0e6e8461eb Author: Theodore Vasiloudis t...@sics.se Date: 2015-04-27T14:16:18Z Added documentation, some minor fixes. commit 4a2235c4606b03fbee8854dd02dea7faa27ace9b Author: Theodore Vasiloudis t...@sics.se Date: 2015-04-28T13:43:17Z Added license to doc file commit afb281c0273af944fabdca96d833d95a094e7944 Author: Theodore Vasiloudis t...@sics.se Date: 2015-04-29T12:36:12Z Style fixes commit 9a810b70f55fc62164d873678437f995b44f4d8e Author: Theodore Vasiloudis t...@sics.se Date: 2015-05-04T08:50:49Z Refactored the way regularization is applied. We are now using pattern matching to determine if a regularization type is differentiable or not. If it is (L2) we apply the regularization at the gradient calculation step, before taking the update step. If it isn't (L1) the regularization is applied after the gradient descent step has been taken. This sets us up nicely for the L-BFGS algorithm, where we can calculate the regularized loss and gradient required if we are using L2. commit 9c71e1a18f4011fdaec53945308c230ab6a97752 Author: Theodore Vasiloudis t...@sics.se Date: 2015-05-05T08:50:52Z Added option to provide UDF for the prediction function, moved SGD regularization to update step. Incorporated the rest of Till's comments. commit 3a0ef8588290c17e83bc0ffa86f1e54d10bf39e0 Author: Theodore Vasiloudis t...@sics.se Date: 2015-05-05T12:16:50Z Style hotfix commit 8314594d547557886630f3076c8f0a72bb478fac Author: Theodore Vasiloudis t...@sics.se Date: 2015-05-05T12:35:53Z Regularization test check fix commit b8ec680d7833669e19f46c6c69f29b76b82d18f5 Author: Theodore Vasiloudis t...@sics.se Date: 2015-05-06T14:34:08Z Added prediction function class to alow non-linear optimization in the future. Small refactoring to allow calculation of regularized loss separatly from regularized gradient. commit 63115fbeff237642e1be87f143cb5042a4aeeff7 Author: Johannes Günther jguenth1 Date: 2015-05-07T09:46:24Z Implemented Hinge Loss commit aca48e33b4cb9c90006a77caddc5fa8f8c057217 Author: mguldner mathieuguldner@gmail.com Date: 2015-05-07T09:49:12Z Add LogisticLoss Function Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530425#comment-14530425 ] Johannes Günther commented on FLINK-1979: - We are two Students part of the DIMA Big data Project and would like to implement this! We would be happy for input concerning input and output of the loss functions Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[
https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530471#comment-14530471
]
Till Rohrmann commented on FLINK-1979:
--
Hi [~jguenther], great to hear that you wanna picks this topic up. We're
currently developing an optimization framework which can tremendously benefit
from more loss functions. See FLINK-1889 and FLINK-1807 for more details.
Here is [https://github.com/apache/flink/pull/613] the corresponding pull
request with the current state. We hope to merge it in the next days. You find
the interface for the loss functions in the file LossFunction.scala. There are
also an implementation for the squared loss function.
Be aware that the prediction function will still change a little bit. But they
will more or less have the following interface
{code:Scala}
trait PredictionFunction {
def predict(x: Vector, weights: WeightVector): Double
def gradient(x: Vector, weights: WeightVector): Vector
}
{code}
So in order to implement new loss functions you simply have to implement the
abstract methods.
Shall I assign this issue to you, Johannes?
Implement Loss Functions
Key: FLINK-1979
URL: https://issues.apache.org/jira/browse/FLINK-1979
Project: Flink
Issue Type: Improvement
Components: Machine Learning Library
Reporter: Johannes Günther
Priority: Minor
Labels: ML
For convex optimization problems, optimizer methods like SGD rely on a
pluggable implementation of a loss function and its first derivative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1979) Implement Loss Functions
[ https://issues.apache.org/jira/browse/FLINK-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530484#comment-14530484 ] Johannes Günther commented on FLINK-1979: - Yes that would be great ! Implement Loss Functions Key: FLINK-1979 URL: https://issues.apache.org/jira/browse/FLINK-1979 Project: Flink Issue Type: Improvement Components: Machine Learning Library Reporter: Johannes Günther Assignee: Johannes Günther Priority: Minor Labels: ML For convex optimization problems, optimizer methods like SGD rely on a pluggable implementation of a loss function and its first derivative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
