szha closed pull request #12293: fixed broken links in logistic regression tutorial URL: https://github.com/apache/incubator-mxnet/pull/12293
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/tutorials/gluon/logistic_regression_explained.md b/docs/tutorials/gluon/logistic_regression_explained.md index 8e5e4a547a6..577a91413b3 100644 --- a/docs/tutorials/gluon/logistic_regression_explained.md +++ b/docs/tutorials/gluon/logistic_regression_explained.md @@ -55,9 +55,9 @@ val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True) ## Defining and training the model -The only requirement for the logistic regression is that the last layer of the network must be a single neuron. Apache MXNet allows us to do so by using [Dense](https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Dense) layer and specifying the number of units to 1. The rest of the network can be arbitrarily complex. +The only requirement for the logistic regression is that the last layer of the network must be a single neuron. Apache MXNet allows us to do so by using [Dense](https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Dense) layer and specifying the number of units to 1. The rest of the network can be arbitrarily complex. -Below, we define a model which has an input layer of 10 neurons, a couple of inner layers of 10 neurons each, and output layer of 1 neuron. We stack the layers using [HybridSequential](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.HybridSequential) block and initialize parameters of the network using [Xavier](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initialization. +Below, we define a model which has an input layer of 10 neurons, a couple of inner layers of 10 neurons each, and output layer of 1 neuron. We stack the layers using [HybridSequential](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.HybridSequential) block and initialize parameters of the network using [Xavier](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initialization. ```python @@ -78,14 +78,14 @@ Loss function is used to calculate how the output of the network differs from th Trainer object allows to specify the method of training to be used. For our tutorial we use [Stochastic Gradient Descent (SGD)](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.optimizer.SGD). For more information on SGD refer to [the following tutorial](https://gluon.mxnet.io/chapter06_optimization/gd-sgd-scratch.html). We also need to parametrize it with learning rate value, which defines the weight updates, and weight decay, which is used for regularization. -Metric helps us to estimate how good our model is in terms of a problem we are trying to solve. Where loss function has more importance for the training process, a metric is usually the thing we are trying to improve and reach maximum value. We also can use more than one metric, to measure various aspects of our model. In our example, we are using [Accuracy](https://mxnet.incubator.apache.org/api/python/model.html#mxnet.metric.Accuracy) and [F1 score](https://mxnet.incubator.apache.org/api/python/model.html#mxnet.metric.F1) as measurements of success of our model. +Metric helps us to estimate how good our model is in terms of a problem we are trying to solve. Where loss function has more importance for the training process, a metric is usually the thing we are trying to improve and reach maximum value. We also can use more than one metric, to measure various aspects of our model. In our example, we are using [Accuracy](https://mxnet.incubator.apache.org/api/python/metric/metric.html?highlight=metric.acc#mxnet.metric.Accuracy) and [F1 score](http://mxnet.incubator.apache.org/api/python/metric/metric.html?highlight=metric.f1#mxnet.metric.F1) as measurements of success of our model. Below we define these objects. ```python loss = gluon.loss.SigmoidBinaryCrossEntropyLoss() -trainer = Trainer(params=net.collect_params(), optimizer='sgd', +trainer = Trainer(params=net.collect_params(), optimizer='sgd', optimizer_params={'learning_rate': 0.1}) accuracy = mx.metric.Accuracy() f1 = mx.metric.F1() @@ -97,16 +97,16 @@ The next step is to define the training function in which we iterate over all ba ```python def train_model(): cumulative_train_loss = 0 - + for i, (data, label) in enumerate(train_dataloader): with autograd.record(): # Do forward pass on a batch of training data output = net(data) - + # Calculate loss for the training data batch loss_result = loss(output, label) - # Calculate gradients + # Calculate gradients loss_result.backward() # Update parameters of the network @@ -114,15 +114,15 @@ def train_model(): # sum losses of every batch cumulative_train_loss += nd.sum(loss_result).asscalar() - + return cumulative_train_loss ``` ## Validating the model -Our validation function is very similar to the training one. The main difference is that we want to calculate accuracy of the model. We use [Accuracy metric](https://mxnet.incubator.apache.org/api/python/model.html#mxnet.metric.Accuracy) to do so. +Our validation function is very similar to the training one. The main difference is that we want to calculate accuracy of the model. We use [Accuracy metric](https://mxnet.incubator.apache.org/api/python/metric/metric.html?highlight=metric.acc#mxnet.metric.Accuracy) to do so. -`Accuracy` metric requires 2 arguments: 1) a vector of ground-truth classes and 2) A vector or matrix of predictions. When predictions are of the same shape as the vector of ground-truth classes, `Accuracy` class assumes that prediction vector contains predicted classes. So, it converts the vector to `Int32` and compare each item of ground-truth classes to prediction vector. +`Accuracy` metric requires 2 arguments: 1) a vector of ground-truth classes and 2) A vector or matrix of predictions. When predictions are of the same shape as the vector of ground-truth classes, `Accuracy` class assumes that prediction vector contains predicted classes. So, it converts the vector to `Int32` and compare each item of ground-truth classes to prediction vector. Because of the behaviour above, you will get an unexpected result if you just apply [Sigmoid](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.sigmoid) function to the network result and pass it to `Accuracy` metric. As mentioned before, we need to apply `Sigmoid` function to the output of the neuron to get a probability of belonging to the class 1. But `Sigmoid` function produces output in range [0; 1], and all numbers in that range are going to be casted to 0, even if it is as high as 0.99. To avoid this we write a custom bit of code on line 12, that: @@ -146,27 +146,27 @@ Then we pass this stacked matrix to `F1` score. ```python def validate_model(threshold): cumulative_val_loss = 0 - + for i, (val_data, val_ground_truth_class) in enumerate(val_dataloader): # Do forward pass on a batch of validation data output = net(val_data) - + # Similar to cumulative training loss, calculate cumulative validation loss cumulative_val_loss += nd.sum(loss(output, val_ground_truth_class)).asscalar() - + # getting prediction as a sigmoid prediction = net(val_data).sigmoid() - + # Converting neuron outputs to classes predicted_classes = mx.nd.ceil(prediction - threshold) - + # Update validation accuracy - accuracy.update(val_ground_truth_class, predicted_classes.reshape(-1)) - + accuracy.update(val_ground_truth_class, predicted_classes.reshape(-1)) + # calculate probabilities of belonging to different classes. F1 metric works only with this notation prediction = prediction.reshape(-1) probabilities = mx.nd.stack(1 - prediction, prediction, axis=1) - + f1.update(val_ground_truth_class, probabilities) return cumulative_val_loss @@ -184,8 +184,8 @@ threshold = 0.5 for e in range(epochs): avg_train_loss = train_model() / train_data_size avg_val_loss = validate_model(threshold) / val_data_size - - print("Epoch: %s, Training loss: %.2f, Validation loss: %.2f, Validation accuracy: %.2f, F1 score: %.2f" % + + print("Epoch: %s, Training loss: %.2f, Validation loss: %.2f, Validation accuracy: %.2f, F1 score: %.2f" % (e, avg_train_loss, avg_val_loss, accuracy.get()[1], f1.get()[1])) # we reset accuracy, so the new epoch's accuracy would be calculated from the blank state @@ -203,13 +203,13 @@ for e in range(epochs): Epoch: 4, Training loss: 0.06, Validation loss: 0.09, Validation accuracy: 0.97, F1 score: 0.58 <!--notebook-skip-line--> Epoch: 5, Training loss: 0.04, Validation loss: 0.12, Validation accuracy: 0.97, F1 score: 0.59 <!--notebook-skip-line--> - + Epoch: 6, Training loss: 0.05, Validation loss: 0.09, Validation accuracy: 0.99, F1 score: 0.62 <!--notebook-skip-line--> - + Epoch: 7, Training loss: 0.05, Validation loss: 0.10, Validation accuracy: 0.97, F1 score: 0.62 <!--notebook-skip-line--> - + Epoch: 8, Training loss: 0.05, Validation loss: 0.12, Validation accuracy: 0.95, F1 score: 0.63 <!--notebook-skip-line--> - + Epoch: 9, Training loss: 0.04, Validation loss: 0.09, Validation accuracy: 0.98, F1 score: 0.65 <!--notebook-skip-line--> @@ -217,7 +217,7 @@ In our case we hit the accuracy of 0.98 and F1 score of 0.65. ## Tip 1: Use only one neuron in the output layer -Despite that there are 2 classes, there should be only one output neuron, because `SigmoidBinaryCrossEntropyLoss` accepts only one feature as an input. +Despite that there are 2 classes, there should be only one output neuron, because `SigmoidBinaryCrossEntropyLoss` accepts only one feature as an input. ## Tip 2: Encode classes as 0 and 1 @@ -225,7 +225,7 @@ For `SigmoidBinaryCrossEntropyLoss` to work it is required that classes were enc ## Tip 3: Use SigmoidBinaryCrossEntropyLoss instead of LogisticRegressionOutput -NDArray API has two options to calculate logistic regression loss: [SigmoidBinaryCrossEntropyLoss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss) and [LogisticRegressionOutput](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.LogisticRegressionOutput). `LogisticRegressionOutput` is designed to be an output layer when using the Module API, and is not supposed to be used when using Gluon API. +NDArray API has two options to calculate logistic regression loss: [SigmoidBinaryCrossEntropyLoss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss) and [LogisticRegressionOutput](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.LogisticRegressionOutput). `LogisticRegressionOutput` is designed to be an output layer when using the Module API, and is not supposed to be used when using Gluon API. ## Conclusion @@ -235,4 +235,4 @@ In this tutorial I explained some potential pitfalls to be aware of. When doing 1. Use `SigmoidBinaryCrossEntropyLoss` 1. Convert probabilities to classes before calculating Accuracy -<!-- INSERT SOURCE DOWNLOAD BUTTONS --> \ No newline at end of file +<!-- INSERT SOURCE DOWNLOAD BUTTONS --> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
