http://git-wip-us.apache.org/repos/asf/incubator-hivemall-site/blob/a98b42f8/userguide/eval/binary_classification_measures.html ---------------------------------------------------------------------- diff --git a/userguide/eval/binary_classification_measures.html b/userguide/eval/binary_classification_measures.html new file mode 100644 index 0000000..75e9186 --- /dev/null +++ b/userguide/eval/binary_classification_measures.html @@ -0,0 +1,2514 @@ + +<!DOCTYPE HTML> +<html lang="" > + <head> + <meta charset="UTF-8"> + <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> + <title>Binary Classification Metrics · Hivemall User Manual</title> + <meta http-equiv="X-UA-Compatible" content="IE=edge" /> + <meta name="description" content=""> + <meta name="generator" content="GitBook 3.2.2"> + + + + + <link rel="stylesheet" href="../gitbook/style.css"> + + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-splitter/splitter.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-etoc/plugin.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-callouts/plugin.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-toggle-chapters/toggle.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-codeblock-filename/block.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-expandable-chapters/expandable-chapters.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-multipart/multipart.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-katex/katex.min.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-emphasize/plugin.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-highlight/website.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-search/search.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-fontsettings/website.css"> + + + + <link rel="stylesheet" href="../gitbook/gitbook-plugin-theme-api/theme-api.css"> + + + + + + + + + + + + + + + + + + + + + + + + <meta name="HandheldFriendly" content="true"/> + <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> + <meta name="apple-mobile-web-app-capable" content="yes"> + <meta name="apple-mobile-web-app-status-bar-style" content="black"> + <link rel="apple-touch-icon-precomposed" sizes="152x152" href="../gitbook/images/apple-touch-icon-precomposed-152.png"> + <link rel="shortcut icon" href="../gitbook/images/favicon.ico" type="image/x-icon"> + + + <link rel="next" href="auc.html" /> + + + <link rel="prev" href="../ft_engineering/tfidf.html" /> + + + </head> + <body> + +<div class="book"> + <div class="book-summary"> + + +<div id="book-search-input" role="search"> + <input type="text" placeholder="Type to search" /> +</div> + + + <nav role="navigation"> + + + +<ul class="summary"> + + + + + <li> + <a href="http://hivemall.incubator.apache.org/"; target="_blank" class="custom-link"><i class="fa fa-home"></i> Home</a> + </li> + + + + + <li class="divider"></li> + + + + + <li class="header">TABLE OF CONTENTS</li> + + + + <li class="chapter " data-level="1.1" data-path="../"> + + <a href="../"> + + + <b>1.1.</b> + + Introduction + + </a> + + + + </li> + + <li class="chapter " data-level="1.2" data-path="../getting_started/"> + + <a href="../getting_started/"> + + + <b>1.2.</b> + + Getting Started + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="1.2.1" data-path="../getting_started/installation.html"> + + <a href="../getting_started/installation.html"> + + + <b>1.2.1.</b> + + Installation + + </a> + + + + </li> + + <li class="chapter " data-level="1.2.2" data-path="../getting_started/permanent-functions.html"> + + <a href="../getting_started/permanent-functions.html"> + + + <b>1.2.2.</b> + + Install as permanent functions + + </a> + + + + </li> + + <li class="chapter " data-level="1.2.3" data-path="../getting_started/input-format.html"> + + <a href="../getting_started/input-format.html"> + + + <b>1.2.3.</b> + + Input Format + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="1.3" data-path="../tips/"> + + <a href="../tips/"> + + + <b>1.3.</b> + + Tips for Effective Hivemall + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="1.3.1" data-path="../tips/addbias.html"> + + <a href="../tips/addbias.html"> + + + <b>1.3.1.</b> + + Explicit add_bias() for better prediction + + </a> + + + + </li> + + <li class="chapter " data-level="1.3.2" data-path="../tips/rand_amplify.html"> + + <a href="../tips/rand_amplify.html"> + + + <b>1.3.2.</b> + + Use rand_amplify() to better prediction results + + </a> + + + + </li> + + <li class="chapter " data-level="1.3.3" data-path="../tips/rt_prediction.html"> + + <a href="../tips/rt_prediction.html"> + + + <b>1.3.3.</b> + + Real-time Prediction on RDBMS + + </a> + + + + </li> + + <li class="chapter " data-level="1.3.4" data-path="../tips/ensemble_learning.html"> + + <a href="../tips/ensemble_learning.html"> + + + <b>1.3.4.</b> + + Ensemble learning for stable prediction + + </a> + + + + </li> + + <li class="chapter " data-level="1.3.5" data-path="../tips/mixserver.html"> + + <a href="../tips/mixserver.html"> + + + <b>1.3.5.</b> + + Mixing models for a better prediction convergence (MIX server) + + </a> + + + + </li> + + <li class="chapter " data-level="1.3.6" data-path="../tips/emr.html"> + + <a href="../tips/emr.html"> + + + <b>1.3.6.</b> + + Run Hivemall on Amazon Elastic MapReduce + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="1.4" data-path="../tips/general_tips.html"> + + <a href="../tips/general_tips.html"> + + + <b>1.4.</b> + + General Hive/Hadoop tips + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="1.4.1" data-path="../tips/rowid.html"> + + <a href="../tips/rowid.html"> + + + <b>1.4.1.</b> + + Adding rowid for each row + + </a> + + + + </li> + + <li class="chapter " data-level="1.4.2" data-path="../tips/hadoop_tuning.html"> + + <a href="../tips/hadoop_tuning.html"> + + + <b>1.4.2.</b> + + Hadoop tuning for Hivemall + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="1.5" data-path="../troubleshooting/"> + + <a href="../troubleshooting/"> + + + <b>1.5.</b> + + Troubleshooting + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="1.5.1" data-path="../troubleshooting/oom.html"> + + <a href="../troubleshooting/oom.html"> + + + <b>1.5.1.</b> + + OutOfMemoryError in training + + </a> + + + + </li> + + <li class="chapter " data-level="1.5.2" data-path="../troubleshooting/mapjoin_task_error.html"> + + <a href="../troubleshooting/mapjoin_task_error.html"> + + + <b>1.5.2.</b> + + SemanticException Generate Map Join Task Error: Cannot serialize object + + </a> + + + + </li> + + <li class="chapter " data-level="1.5.3" data-path="../troubleshooting/asterisk.html"> + + <a href="../troubleshooting/asterisk.html"> + + + <b>1.5.3.</b> + + Asterisk argument for UDTF does not work + + </a> + + + + </li> + + <li class="chapter " data-level="1.5.4" data-path="../troubleshooting/num_mappers.html"> + + <a href="../troubleshooting/num_mappers.html"> + + + <b>1.5.4.</b> + + The number of mappers is less than input splits in Hadoop 2.x + + </a> + + + + </li> + + <li class="chapter " data-level="1.5.5" data-path="../troubleshooting/mapjoin_classcastex.html"> + + <a href="../troubleshooting/mapjoin_classcastex.html"> + + + <b>1.5.5.</b> + + Map-side Join causes ClassCastException on Tez + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part II - Generic Features</li> + + + + <li class="chapter " data-level="2.1" data-path="../misc/generic_funcs.html"> + + <a href="../misc/generic_funcs.html"> + + + <b>2.1.</b> + + List of generic Hivemall functions + + </a> + + + + </li> + + <li class="chapter " data-level="2.2" data-path="../misc/topk.html"> + + <a href="../misc/topk.html"> + + + <b>2.2.</b> + + Efficient Top-K query processing + + </a> + + + + </li> + + <li class="chapter " data-level="2.3" data-path="../misc/tokenizer.html"> + + <a href="../misc/tokenizer.html"> + + + <b>2.3.</b> + + Text Tokenizer + + </a> + + + + </li> + + + + + <li class="header">Part III - Feature Engineering</li> + + + + <li class="chapter " data-level="3.1" data-path="../ft_engineering/scaling.html"> + + <a href="../ft_engineering/scaling.html"> + + + <b>3.1.</b> + + Feature Scaling + + </a> + + + + </li> + + <li class="chapter " data-level="3.2" data-path="../ft_engineering/hashing.html"> + + <a href="../ft_engineering/hashing.html"> + + + <b>3.2.</b> + + Feature Hashing + + </a> + + + + </li> + + <li class="chapter " data-level="3.3" data-path="../ft_engineering/selection.html"> + + <a href="../ft_engineering/selection.html"> + + + <b>3.3.</b> + + Feature Selection + + </a> + + + + </li> + + <li class="chapter " data-level="3.4" data-path="../ft_engineering/binning.html"> + + <a href="../ft_engineering/binning.html"> + + + <b>3.4.</b> + + Feature Binning + + </a> + + + + </li> + + <li class="chapter " data-level="3.5" data-path="../ft_engineering/pairing.html"> + + <a href="../ft_engineering/pairing.html"> + + + <b>3.5.</b> + + FEATURE PAIRING + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="3.5.1" data-path="../ft_engineering/polynomial.html"> + + <a href="../ft_engineering/polynomial.html"> + + + <b>3.5.1.</b> + + Polynomial Features + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="3.6" data-path="../ft_engineering/ft_trans.html"> + + <a href="../ft_engineering/ft_trans.html"> + + + <b>3.6.</b> + + FEATURE TRANSFORMATION + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="3.6.1" data-path="../ft_engineering/vectorization.html"> + + <a href="../ft_engineering/vectorization.html"> + + + <b>3.6.1.</b> + + Feature Vectorization + + </a> + + + + </li> + + <li class="chapter " data-level="3.6.2" data-path="../ft_engineering/quantify.html"> + + <a href="../ft_engineering/quantify.html"> + + + <b>3.6.2.</b> + + Quantify non-number features + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="3.7" data-path="../ft_engineering/tfidf.html"> + + <a href="../ft_engineering/tfidf.html"> + + + <b>3.7.</b> + + TF-IDF Calculation + + </a> + + + + </li> + + + + + <li class="header">Part IV - Evaluation</li> + + + + <li class="chapter active" data-level="4.1" data-path="binary_classification_measures.html"> + + <a href="binary_classification_measures.html"> + + + <b>4.1.</b> + + Binary Classification Metrics + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="4.1.1" data-path="auc.html"> + + <a href="auc.html"> + + + <b>4.1.1.</b> + + Area Under the ROC Curve + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="4.2" data-path="multilabel_classification_measures.html"> + + <a href="multilabel_classification_measures.html"> + + + <b>4.2.</b> + + Multi-label Classification Metrics + + </a> + + + + </li> + + <li class="chapter " data-level="4.3" data-path="regression.html"> + + <a href="regression.html"> + + + <b>4.3.</b> + + Regression metrics + + </a> + + + + </li> + + <li class="chapter " data-level="4.4" data-path="rank.html"> + + <a href="rank.html"> + + + <b>4.4.</b> + + Ranking Measures + + </a> + + + + </li> + + <li class="chapter " data-level="4.5" data-path="datagen.html"> + + <a href="datagen.html"> + + + <b>4.5.</b> + + Data Generation + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="4.5.1" data-path="lr_datagen.html"> + + <a href="lr_datagen.html"> + + + <b>4.5.1.</b> + + Logistic Regression data generation + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part V - Supervised Learning</li> + + + + <li class="chapter " data-level="5.1" data-path="../misc/prediction.html"> + + <a href="../misc/prediction.html"> + + + <b>5.1.</b> + + How Prediction Works + + </a> + + + + </li> + + + + + <li class="header">Part VI - Binary classification</li> + + + + <li class="chapter " data-level="6.1" data-path="../binaryclass/general.html"> + + <a href="../binaryclass/general.html"> + + + <b>6.1.</b> + + Binary Classification + + </a> + + + + </li> + + <li class="chapter " data-level="6.2" data-path="../binaryclass/a9a.html"> + + <a href="../binaryclass/a9a.html"> + + + <b>6.2.</b> + + a9a tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="6.2.1" data-path="../binaryclass/a9a_dataset.html"> + + <a href="../binaryclass/a9a_dataset.html"> + + + <b>6.2.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="6.2.2" data-path="../binaryclass/a9a_lr.html"> + + <a href="../binaryclass/a9a_lr.html"> + + + <b>6.2.2.</b> + + Logistic Regression + + </a> + + + + </li> + + <li class="chapter " data-level="6.2.3" data-path="../binaryclass/a9a_minibatch.html"> + + <a href="../binaryclass/a9a_minibatch.html"> + + + <b>6.2.3.</b> + + Mini-batch Gradient Descent + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="6.3" data-path="../binaryclass/news20.html"> + + <a href="../binaryclass/news20.html"> + + + <b>6.3.</b> + + News20 tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="6.3.1" data-path="../binaryclass/news20_dataset.html"> + + <a href="../binaryclass/news20_dataset.html"> + + + <b>6.3.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="6.3.2" data-path="../binaryclass/news20_pa.html"> + + <a href="../binaryclass/news20_pa.html"> + + + <b>6.3.2.</b> + + Perceptron, Passive Aggressive + + </a> + + + + </li> + + <li class="chapter " data-level="6.3.3" data-path="../binaryclass/news20_scw.html"> + + <a href="../binaryclass/news20_scw.html"> + + + <b>6.3.3.</b> + + CW, AROW, SCW + + </a> + + + + </li> + + <li class="chapter " data-level="6.3.4" data-path="../binaryclass/news20_adagrad.html"> + + <a href="../binaryclass/news20_adagrad.html"> + + + <b>6.3.4.</b> + + AdaGradRDA, AdaGrad, AdaDelta + + </a> + + + + </li> + + <li class="chapter " data-level="6.3.5" data-path="../binaryclass/news20_rf.html"> + + <a href="../binaryclass/news20_rf.html"> + + + <b>6.3.5.</b> + + Random Forest + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="6.4" data-path="../binaryclass/kdd2010a.html"> + + <a href="../binaryclass/kdd2010a.html"> + + + <b>6.4.</b> + + KDD2010a tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="6.4.1" data-path="../binaryclass/kdd2010a_dataset.html"> + + <a href="../binaryclass/kdd2010a_dataset.html"> + + + <b>6.4.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="6.4.2" data-path="../binaryclass/kdd2010a_scw.html"> + + <a href="../binaryclass/kdd2010a_scw.html"> + + + <b>6.4.2.</b> + + PA, CW, AROW, SCW + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="6.5" data-path="../binaryclass/kdd2010b.html"> + + <a href="../binaryclass/kdd2010b.html"> + + + <b>6.5.</b> + + KDD2010b tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="6.5.1" data-path="../binaryclass/kdd2010b_dataset.html"> + + <a href="../binaryclass/kdd2010b_dataset.html"> + + + <b>6.5.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="6.5.2" data-path="../binaryclass/kdd2010b_arow.html"> + + <a href="../binaryclass/kdd2010b_arow.html"> + + + <b>6.5.2.</b> + + AROW + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="6.6" data-path="../binaryclass/webspam.html"> + + <a href="../binaryclass/webspam.html"> + + + <b>6.6.</b> + + Webspam tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="6.6.1" data-path="../binaryclass/webspam_dataset.html"> + + <a href="../binaryclass/webspam_dataset.html"> + + + <b>6.6.1.</b> + + Data pareparation + + </a> + + + + </li> + + <li class="chapter " data-level="6.6.2" data-path="../binaryclass/webspam_scw.html"> + + <a href="../binaryclass/webspam_scw.html"> + + + <b>6.6.2.</b> + + PA1, AROW, SCW + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="6.7" data-path="../binaryclass/titanic_rf.html"> + + <a href="../binaryclass/titanic_rf.html"> + + + <b>6.7.</b> + + Kaggle Titanic tutorial + + </a> + + + + </li> + + + + + <li class="header">Part VII - Multiclass classification</li> + + + + <li class="chapter " data-level="7.1" data-path="../multiclass/news20.html"> + + <a href="../multiclass/news20.html"> + + + <b>7.1.</b> + + News20 Multiclass tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="7.1.1" data-path="../multiclass/news20_dataset.html"> + + <a href="../multiclass/news20_dataset.html"> + + + <b>7.1.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="7.1.2" data-path="../multiclass/news20_one-vs-the-rest_dataset.html"> + + <a href="../multiclass/news20_one-vs-the-rest_dataset.html"> + + + <b>7.1.2.</b> + + Data preparation for one-vs-the-rest classifiers + + </a> + + + + </li> + + <li class="chapter " data-level="7.1.3" data-path="../multiclass/news20_pa.html"> + + <a href="../multiclass/news20_pa.html"> + + + <b>7.1.3.</b> + + PA + + </a> + + + + </li> + + <li class="chapter " data-level="7.1.4" data-path="../multiclass/news20_scw.html"> + + <a href="../multiclass/news20_scw.html"> + + + <b>7.1.4.</b> + + CW, AROW, SCW + + </a> + + + + </li> + + <li class="chapter " data-level="7.1.5" data-path="../multiclass/news20_ensemble.html"> + + <a href="../multiclass/news20_ensemble.html"> + + + <b>7.1.5.</b> + + Ensemble learning + + </a> + + + + </li> + + <li class="chapter " data-level="7.1.6" data-path="../multiclass/news20_one-vs-the-rest.html"> + + <a href="../multiclass/news20_one-vs-the-rest.html"> + + + <b>7.1.6.</b> + + one-vs-the-rest classifier + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="7.2" data-path="../multiclass/iris.html"> + + <a href="../multiclass/iris.html"> + + + <b>7.2.</b> + + Iris tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="7.2.1" data-path="../multiclass/iris_dataset.html"> + + <a href="../multiclass/iris_dataset.html"> + + + <b>7.2.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="7.2.2" data-path="../multiclass/iris_scw.html"> + + <a href="../multiclass/iris_scw.html"> + + + <b>7.2.2.</b> + + SCW + + </a> + + + + </li> + + <li class="chapter " data-level="7.2.3" data-path="../multiclass/iris_randomforest.html"> + + <a href="../multiclass/iris_randomforest.html"> + + + <b>7.2.3.</b> + + Random Forest + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part VIII - Regression</li> + + + + <li class="chapter " data-level="8.1" data-path="../regression/general.html"> + + <a href="../regression/general.html"> + + + <b>8.1.</b> + + Regression + + </a> + + + + </li> + + <li class="chapter " data-level="8.2" data-path="../regression/e2006.html"> + + <a href="../regression/e2006.html"> + + + <b>8.2.</b> + + E2006-tfidf regression tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="8.2.1" data-path="../regression/e2006_dataset.html"> + + <a href="../regression/e2006_dataset.html"> + + + <b>8.2.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="8.2.2" data-path="../regression/e2006_arow.html"> + + <a href="../regression/e2006_arow.html"> + + + <b>8.2.2.</b> + + Passive Aggressive, AROW + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="8.3" data-path="../regression/kddcup12tr2.html"> + + <a href="../regression/kddcup12tr2.html"> + + + <b>8.3.</b> + + KDDCup 2012 track 2 CTR prediction tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="8.3.1" data-path="../regression/kddcup12tr2_dataset.html"> + + <a href="../regression/kddcup12tr2_dataset.html"> + + + <b>8.3.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="8.3.2" data-path="../regression/kddcup12tr2_lr.html"> + + <a href="../regression/kddcup12tr2_lr.html"> + + + <b>8.3.2.</b> + + Logistic Regression, Passive Aggressive + + </a> + + + + </li> + + <li class="chapter " data-level="8.3.3" data-path="../regression/kddcup12tr2_lr_amplify.html"> + + <a href="../regression/kddcup12tr2_lr_amplify.html"> + + + <b>8.3.3.</b> + + Logistic Regression with Amplifier + + </a> + + + + </li> + + <li class="chapter " data-level="8.3.4" data-path="../regression/kddcup12tr2_adagrad.html"> + + <a href="../regression/kddcup12tr2_adagrad.html"> + + + <b>8.3.4.</b> + + AdaGrad, AdaDelta + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part IX - Recommendation</li> + + + + <li class="chapter " data-level="9.1" data-path="../recommend/cf.html"> + + <a href="../recommend/cf.html"> + + + <b>9.1.</b> + + Collaborative Filtering + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="9.1.1" data-path="../recommend/item_based_cf.html"> + + <a href="../recommend/item_based_cf.html"> + + + <b>9.1.1.</b> + + Item-based Collaborative Filtering + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="9.2" data-path="../recommend/news20.html"> + + <a href="../recommend/news20.html"> + + + <b>9.2.</b> + + News20 related article recommendation Tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="9.2.1" data-path="../multiclass/news20_dataset.html"> + + <a href="../multiclass/news20_dataset.html"> + + + <b>9.2.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="9.2.2" data-path="../recommend/news20_jaccard.html"> + + <a href="../recommend/news20_jaccard.html"> + + + <b>9.2.2.</b> + + LSH/Minhash and Jaccard Similarity + + </a> + + + + </li> + + <li class="chapter " data-level="9.2.3" data-path="../recommend/news20_knn.html"> + + <a href="../recommend/news20_knn.html"> + + + <b>9.2.3.</b> + + LSH/Minhash and Brute-Force Search + + </a> + + + + </li> + + <li class="chapter " data-level="9.2.4" data-path="../recommend/news20_bbit_minhash.html"> + + <a href="../recommend/news20_bbit_minhash.html"> + + + <b>9.2.4.</b> + + kNN search using b-Bits Minhash + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="9.3" data-path="../recommend/movielens.html"> + + <a href="../recommend/movielens.html"> + + + <b>9.3.</b> + + MovieLens movie recommendation Tutorial + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="9.3.1" data-path="../recommend/movielens_dataset.html"> + + <a href="../recommend/movielens_dataset.html"> + + + <b>9.3.1.</b> + + Data preparation + + </a> + + + + </li> + + <li class="chapter " data-level="9.3.2" data-path="../recommend/movielens_cf.html"> + + <a href="../recommend/movielens_cf.html"> + + + <b>9.3.2.</b> + + Item-based Collaborative Filtering + + </a> + + + + </li> + + <li class="chapter " data-level="9.3.3" data-path="../recommend/movielens_mf.html"> + + <a href="../recommend/movielens_mf.html"> + + + <b>9.3.3.</b> + + Matrix Factorization + + </a> + + + + </li> + + <li class="chapter " data-level="9.3.4" data-path="../recommend/movielens_fm.html"> + + <a href="../recommend/movielens_fm.html"> + + + <b>9.3.4.</b> + + Factorization Machine + + </a> + + + + </li> + + <li class="chapter " data-level="9.3.5" data-path="../recommend/movielens_cv.html"> + + <a href="../recommend/movielens_cv.html"> + + + <b>9.3.5.</b> + + 10-fold Cross Validation (Matrix Factorization) + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part X - Anomaly Detection</li> + + + + <li class="chapter " data-level="10.1" data-path="../anomaly/lof.html"> + + <a href="../anomaly/lof.html"> + + + <b>10.1.</b> + + Outlier Detection using Local Outlier Factor (LOF) + + </a> + + + + </li> + + <li class="chapter " data-level="10.2" data-path="../anomaly/sst.html"> + + <a href="../anomaly/sst.html"> + + + <b>10.2.</b> + + Change-Point Detection using Singular Spectrum Transformation (SST) + + </a> + + + + </li> + + <li class="chapter " data-level="10.3" data-path="../anomaly/changefinder.html"> + + <a href="../anomaly/changefinder.html"> + + + <b>10.3.</b> + + ChangeFinder: Detecting Outlier and Change-Point Simultaneously + + </a> + + + + </li> + + + + + <li class="header">Part XI - Clustering</li> + + + + <li class="chapter " data-level="11.1" data-path="../clustering/lda.html"> + + <a href="../clustering/lda.html"> + + + <b>11.1.</b> + + Latent Dirichlet Allocation + + </a> + + + + </li> + + <li class="chapter " data-level="11.2" data-path="../clustering/plsa.html"> + + <a href="../clustering/plsa.html"> + + + <b>11.2.</b> + + Probabilistic Latent Semantic Analysis + + </a> + + + + </li> + + + + + <li class="header">Part XII - GeoSpatial functions</li> + + + + <li class="chapter " data-level="12.1" data-path="../geospatial/latlon.html"> + + <a href="../geospatial/latlon.html"> + + + <b>12.1.</b> + + Lat/Lon functions + + </a> + + + + </li> + + + + + <li class="header">Part XIII - Hivemall on Spark</li> + + + + <li class="chapter " data-level="13.1" data-path="../spark/getting_started/"> + + <a href="../spark/getting_started/"> + + + <b>13.1.</b> + + Getting Started + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="13.1.1" data-path="../spark/getting_started/installation.html"> + + <a href="../spark/getting_started/installation.html"> + + + <b>13.1.1.</b> + + Installation + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="13.2" data-path="../spark/binaryclass/"> + + <a href="../spark/binaryclass/"> + + + <b>13.2.</b> + + Binary Classification + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="13.2.1" data-path="../spark/binaryclass/a9a_df.html"> + + <a href="../spark/binaryclass/a9a_df.html"> + + + <b>13.2.1.</b> + + a9a Tutorial for DataFrame + + </a> + + + + </li> + + <li class="chapter " data-level="13.2.2" data-path="../spark/binaryclass/a9a_sql.html"> + + <a href="../spark/binaryclass/a9a_sql.html"> + + + <b>13.2.2.</b> + + a9a Tutorial for SQL + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="13.3" data-path="../spark/binaryclass/"> + + <a href="../spark/binaryclass/"> + + + <b>13.3.</b> + + Regression + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="13.3.1" data-path="../spark/regression/e2006_df.html"> + + <a href="../spark/regression/e2006_df.html"> + + + <b>13.3.1.</b> + + E2006-tfidf regression Tutorial for DataFrame + + </a> + + + + </li> + + <li class="chapter " data-level="13.3.2" data-path="../spark/regression/e2006_sql.html"> + + <a href="../spark/regression/e2006_sql.html"> + + + <b>13.3.2.</b> + + E2006-tfidf regression Tutorial for SQL + + </a> + + + + </li> + + + </ul> + + </li> + + <li class="chapter " data-level="13.4" data-path="../spark/misc/misc.html"> + + <a href="../spark/misc/misc.html"> + + + <b>13.4.</b> + + Generic features + + </a> + + + + <ul class="articles"> + + + <li class="chapter " data-level="13.4.1" data-path="../spark/misc/topk_join.html"> + + <a href="../spark/misc/topk_join.html"> + + + <b>13.4.1.</b> + + Top-k Join processing + + </a> + + + + </li> + + <li class="chapter " data-level="13.4.2" data-path="../spark/misc/functions.html"> + + <a href="../spark/misc/functions.html"> + + + <b>13.4.2.</b> + + Other utility functions + + </a> + + + + </li> + + + </ul> + + </li> + + + + + <li class="header">Part XIV - Hivemall on Docker</li> + + + + <li class="chapter " data-level="14.1" data-path="../docker/getting_started.html"> + + <a href="../docker/getting_started.html"> + + + <b>14.1.</b> + + Getting Started + + </a> + + + + </li> + + + + + <li class="header">Part XIV - External References</li> + + + + <li class="chapter " data-level="15.1" > + + <a target="_blank" href="https://github.com/maropu/hivemall-spark";> + + + <b>15.1.</b> + + Hivemall on Apache Spark + + </a> + + + + </li> + + <li class="chapter " data-level="15.2" > + + <a target="_blank" href="https://github.com/daijyc/hivemall/wiki/PigHome";> + + + <b>15.2.</b> + + Hivemall on Apache Pig + + </a> + + + + </li> + + + + + <li class="divider"></li> + + <li> + <a href="https://www.gitbook.com"; target="blank" class="gitbook-link"> + Published with GitBook + </a> + </li> +</ul> + + + </nav> + + + </div> + + <div class="book-body"> + + <div class="body-inner"> + + + +<div class="book-header" role="navigation"> + + + <!-- Title --> + <h1> + <i class="fa fa-circle-o-notch fa-spin"></i> + <a href=".." >Binary Classification Metrics</a> + </h1> +</div> + + + + + <div class="page-wrapper" tabindex="-1" role="main"> + <div class="page-inner"> + +<div id="book-search-results"> + <div class="search-noresults"> + + <section class="normal markdown-section"> + + <!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +<!-- toc --><div id="toc" class="toc"> + +<ul> +<li><a href="#binary-problems">Binary problems</a></li> +<li><a href="#example">Example</a><ul> +<li><a href="#data">Data</a></li> +<li><a href="#preliminary-metrics">Preliminary metrics</a><ul> +<li><a href="#recall">Recall</a></li> +<li><a href="#precision">Precision</a></li> +</ul> +</li> +</ul> +</li> +<li><a href="#metrics">Metrics</a><ul> +<li><a href="#f1-score">F1-score</a><ul> +<li><a href="#micro-average">Micro average</a></li> +<li><a href="#binary-average">Binary average</a></li> +</ul> +</li> +<li><a href="#f-measure">F-measure</a></li> +</ul> +</li> +</ul> + +</div><!-- tocstop --> +<h1 id="binary-problems">Binary problems</h1> +<p>Binary classification is a task to predict a label of each data given two categories.</p> +<p>Hivemall provides several tutorials to deal with binary classification problems as follows:</p> +<ul> +<li><a href="../binaryclass/general.html">Online advertisement click prediction</a></li> +<li><a href="../binaryclass/news20_dataset.html">News classification</a></li> +</ul> +<p>This page focuses on the evaluation of such binary classification problems. +If your classifier outputs probability rather than 0/1 label, evaluation based on <a href="auc.html">Area Under the ROC Curve</a> would be more appropriate.</p> +<h1 id="example">Example</h1> +<p>This page introduces toy example data and two metrics for explanation.</p> +<h2 id="data">Data</h2> +<p>The following table shows examples of binary classification's prediction.</p> +<table> +<thead> +<tr> +<th style="text-align:center">truth label</th> +<th style="text-align:center">predicted label</th> +<th style="text-align:center">description</th> +</tr> +</thead> +<tbody> +<tr> +<td style="text-align:center">1</td> +<td style="text-align:center">0</td> +<td style="text-align:center">False Negative</td> +</tr> +<tr> +<td style="text-align:center">0</td> +<td style="text-align:center">1</td> +<td style="text-align:center">False Positive</td> +</tr> +<tr> +<td style="text-align:center">0</td> +<td style="text-align:center">0</td> +<td style="text-align:center">True Negative</td> +</tr> +<tr> +<td style="text-align:center">1</td> +<td style="text-align:center">1</td> +<td style="text-align:center">True Positive</td> +</tr> +<tr> +<td style="text-align:center">0</td> +<td style="text-align:center">1</td> +<td style="text-align:center">False Positive</td> +</tr> +<tr> +<td style="text-align:center">0</td> +<td style="text-align:center">0</td> +<td style="text-align:center">True Negative</td> +</tr> +</tbody> +</table> +<p>In this case, <code>1</code> means positive label and <code>0</code> means negative label. +The leftmost column shows truth labels, and center column includes predicted labels.</p> +<h2 id="preliminary-metrics">Preliminary metrics</h2> +<p>Some evaluation metrics are calculated based on 4 values:</p> +<ul> +<li>True Positive (TP): truth label is positive and predicted label is also positive</li> +<li>True Negative (TN): truth label is negative and predicted label is also negative</li> +<li>False Positive (FP): truth label is negative but predicted label is positive</li> +<li>False Negative (FN): truth label is positive but predicted label is negative</li> +</ul> +<p><code>TR</code> and <code>TN</code> represent correct classification, and <code>FP</code> and <code>FN</code> illustrate incorrect ones.</p> +<p>In this example, we can obtain those values:</p> +<ul> +<li>TP: 1</li> +<li>TN: 2</li> +<li>FP: 2</li> +<li>FN: 1</li> +</ul> +<p>if you want to know about those metrics, Wikipedia provides <a href="https://en.wikipedia.org/wiki/Sensitivity_and_specificity"; target="_blank">more detail information</a>.</p> +<h3 id="recall">Recall</h3> +<p>Recall indicates the true positive rate in truth positive labels. +The value is computed by the following equation:</p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow><mo>=</mo><mfrac><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">F</mi><mi mathvariant="normal">N</mi></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{recall} = \frac{\mathrm{\#TP}}{\mathrm{\#TP} + \mathrm{\#FN}} +</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:1.37144em;"></span><span class="strut bottom" style="height:2.25188em;vertical-align:-0.8804400000000001em;"></span><span class="base displaystyle textstyle uncramped"><span class="mord displaystyle textstyle uncramped"><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">a</span><span class="mord mathrm">l</span><span class="mord mathrm">l</span></span><span class="mrel">=</span><span class="mord reset-textstyle displaystyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.6860000000000002em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle cramped"><span class="mord textst yle cramped"><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">F</span><span class="mord mathrm">N</span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.677em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span></span></span></span><span class="baseline-fix"><span class="fo ntsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span></span></p> +<p>In the previous example, <span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mn>2</mn></mrow></mfrac></mrow><annotation encoding="application/x-tex">\mathrm{precision} = \frac{1}{2}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:0.845108em;"></span><span class="strut bottom" style="height:1.190108em;vertical-align:-0.345em;"></span><span class="base textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord mathrm">p</span><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord ma thrm">i</span><span class="mord mathrm">s</span><span class="mord mathrm">i</span><span class="mord mathrm">o</span><span class="mord mathrm">n</span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.345em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped mtight"><span class="mord scriptstyle cramped mtight"><span class="mord mathrm mtight">2</span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.394em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">&# x200B;</span></span><span class="reset-textstyle scriptstyle uncramped mtight"><span class="mord scriptstyle uncramped mtight"><span class="mord mathrm mtight">1</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span>.</p> +<h3 id="precision">Precision</h3> +<p>Precision indicates the true positive rate in positive predictive labels. +The value is computed by the following equation:</p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>=</mo><mfrac><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">F</mi><mi mathvariant="normal">P</mi></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{precision} = \frac{\mathrm{\#TP}}{\mathrm{\#TP} + \mathrm{\#FP}} +</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:1.37144em;"></span><span class="strut bottom" style="height:2.25188em;vertical-align:-0.8804400000000001em;"></span><span class="base displaystyle textstyle uncramped"><span class="mord displaystyle textstyle uncramped"><span class="mord mathrm">p</span><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">i</span><span class="mord mathrm">s</span><span class="mord mathrm">i</span><span class="mord mathrm">o</span><span class="mord mathrm">n</span></span><span class="mrel">=</span><span class="mord reset-textstyle displaystyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.6860000000000002em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-siz e:0em;">​</span></span><span class="reset-textstyle textstyle cramped"><span class="mord textstyle cramped"><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">F</span><span class="mord mathrm">P</span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.677em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><s pan class="mord mathrm">P</span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span></span></p> +<p>In the previous example, <span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mn>3</mn></mrow></mfrac></mrow><annotation encoding="application/x-tex">\mathrm{precision} = \frac{1}{3}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:0.845108em;"></span><span class="strut bottom" style="height:1.190108em;vertical-align:-0.345em;"></span><span class="base textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord mathrm">p</span><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord ma thrm">i</span><span class="mord mathrm">s</span><span class="mord mathrm">i</span><span class="mord mathrm">o</span><span class="mord mathrm">n</span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.345em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped mtight"><span class="mord scriptstyle cramped mtight"><span class="mord mathrm mtight">3</span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.394em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">&# x200B;</span></span><span class="reset-textstyle scriptstyle uncramped mtight"><span class="mord scriptstyle uncramped mtight"><span class="mord mathrm mtight">1</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span>.</p> +<h1 id="metrics">Metrics</h1> +<p>To use metrics examples, please create the following table.</p> +<pre><code class="lang-sql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">table</span> <span class="hljs-keyword">data</span> <span class="hljs-keyword">as</span> + <span class="hljs-keyword">select</span> <span class="hljs-number">1</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> predicted +<span class="hljs-keyword">union</span> all + <span class="hljs-keyword">select</span> <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">1</span> <span class="hljs-keyword">as</span> predicted +<span class="hljs-keyword">union</span> all + <span class="hljs-keyword">select</span> <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> predicted +<span class="hljs-keyword">union</span> all + <span class="hljs-keyword">select</span> <span class="hljs-number">1</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">1</span> <span class="hljs-keyword">as</span> predicted +<span class="hljs-keyword">union</span> all + <span class="hljs-keyword">select</span> <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">1</span> <span class="hljs-keyword">as</span> predicted +<span class="hljs-keyword">union</span> all + <span class="hljs-keyword">select</span> <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> truth, <span class="hljs-number">0</span> <span class="hljs-keyword">as</span> predicted +; +</code></pre> +<h2 id="f1-score">F1-score</h2> +<p>F1-score is the harmonic mean of recall and precision. +F1-score is computed by the following equation:</p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msub><mrow><mi mathvariant="normal">F</mi></mrow><mn>1</mn></msub><mo>=</mo><mn>2</mn><mfrac><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>∗</mo><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="norm al">n</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{F}_1 = 2 \frac{\mathrm{precision} * \mathrm{recall}}{\mathrm{precision} + \mathrm{recall}} +</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:1.37144em;"></span><span class="strut bottom" style="height:2.25188em;vertical-align:-0.8804400000000001em;"></span><span class="base displaystyle textstyle uncramped"><span class="mord"><span class="mord displaystyle textstyle uncramped"><span class="mord mathrm">F</span></span><span class="msupsub"><span class="vlist"><span style="top:0.15em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped mtight"><span class="mord mathrm mtight">1</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span><span class="mrel">=</span><span class="mord mathrm">2</span><span class="mord reset-textstyle displaystyle textstyle uncramped"><span cl ass="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.6860000000000002em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle cramped"><span class="mord textstyle cramped"><span class="mord textstyle cramped"><span class="mord mathrm">p</span><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">i</span><span class="mord mathrm">s</span><span class="mord mathrm">i</span><span class="mord mathrm">o</span><span class="mord mathrm">n</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">a</span><span class="mord mathrm">l</span><span class="mord mathrm">l</span></span></span></span ></span><span style="top:-0.22999999999999998em;"><span >class="fontsize-ensurer reset-size5 size5"><span >style="font-size:0em;">​</span></span><span class="reset-textstyle >textstyle uncramped frac-line"></span></span><span >style="top:-0.677em;"><span class="fontsize-ensurer reset-size5 size5"><span >style="font-size:0em;">​</span></span><span class="reset-textstyle >textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord >textstyle uncramped"><span class="mord mathrm">p</span><span class="mord >mathrm">r</span><span class="mord mathrm">e</span><span class="mord >mathrm">c</span><span class="mord mathrm">i</span><span class="mord >mathrm">s</span><span class="mord mathrm">i</span><span class="mord >mathrm">o</span><span class="mord mathrm">n</span></span><span >class="mbin">∗</span><span class="mord textstyle uncramped"><span >class="mord mathrm">r</span><span class="mord mathrm">e</span><span >class="mord mathrm">c</span><span class="mord mathrm">a</ span><span class="mord mathrm">l</span><span class="mord mathrm">l</span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span></span></p> +<p>Hivemall's <code>fmeasure</code> function provides the option which can switch <code>micro</code>(default) or <code>binary</code> by passing <code>average</code> argument.</p> +<div class="panel panel-warning"><div class="panel-heading"><h3 class="panel-title" id="caution"><i class="fa fa-exclamation-triangle"></i> Caution</h3></div><div class="panel-body"><p>Hivemall also provides <code>f1score</code> function, but it is old function to obtain F1-score. The value of <code>f1score</code> is based on set operation. So, we recommend to use <code>fmeasure</code> function to get F1-score based on this article.</p></div></div> +<p>You can learn more about this from the following external resource:</p> +<ul> +<li><a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"; target="_blank">scikit-learn's F1-score</a></li> +</ul> +<h3 id="micro-average">Micro average</h3> +<p>If <code>micro</code> is passed to <code>average</code>, +recall and precision are modified to consider True Negative. +So, micro f1score are calculated by those modified recall and precision.</p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow><mo>=</mo><mfrac><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">N</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">F</mi><mi mathvariant="normal">N</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">N</mi></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{recall} = \frac{\mathrm{\#TP} + \mathrm{\#TN}}{\mathrm{\#TP} + \mathrm{\#FN} + \mathrm{\#TN}} +</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:1.37144em;"></span><span class="strut bottom" style="height:2.25188em;vertical-align:-0.8804400000000001em;"></span><span class="base displaystyle textstyle uncramped"><span class="mord displaystyle textstyle uncramped"><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">a</span><span class="mord mathrm">l</span><span class="mord mathrm">l</span></span><span class="mrel">=</span><span class="mord reset-textstyle displaystyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.6860000000000002em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle cramped"><span class="mord textst yle cramped"><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">F</span><span class="mord mathrm">N</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">N</span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.677em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord textstyle uncramped" ><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span >class="mord mathrm">P</span></span><span class="mbin">+</span><span >class="mord textstyle uncramped"><span class="mord mathrm">#</span><span >class="mord mathrm">T</span><span class="mord >mathrm">N</span></span></span></span></span><span class="baseline-fix"><span >class="fontsize-ensurer reset-size5 size5"><span >style="font-size:0em;">​</span></span>​</span></span></span><span > class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped >nulldelimiter"></span></span></span></span></span></span></p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>=</mo><mfrac><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">N</mi></mrow></mrow><mrow><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">F</mi><mi mathvariant="normal">P</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">#</mi><mi mathvariant="normal">T</mi><mi mathvariant="normal">N</mi></mrow></mrow>< /mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{precision} = \frac{\mathrm{\#TP} + \mathrm{\#TN}}{\mathrm{\#TP} + \mathrm{\#FP} + \mathrm{\#TN}} +</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="strut" style="height:1.37144em;"></span><span class="strut bottom" style="height:2.25188em;vertical-align:-0.8804400000000001em;"></span><span class="base displaystyle textstyle uncramped"><span class="mord displaystyle textstyle uncramped"><span class="mord mathrm">p</span><span class="mord mathrm">r</span><span class="mord mathrm">e</span><span class="mord mathrm">c</span><span class="mord mathrm">i</span><span class="mord mathrm">s</span><span class="mord mathrm">i</span><span class="mord mathrm">o</span><span class="mord mathrm">n</span></span><span class="mrel">=</span><span class="mord reset-textstyle displaystyle textstyle uncramped"><span class="mopen sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.6860000000000002em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-siz e:0em;">​</span></span><span class="reset-textstyle textstyle cramped"><span class="mord textstyle cramped"><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">F</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle cramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">N</span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.677em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textst yle textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord textstyle uncramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">P</span></span><span class="mbin">+</span><span class="mord textstyle uncramped"><span class="mord mathrm">#</span><span class="mord mathrm">T</span><span class="mord mathrm">N</span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span></span></span></span></span></p> +<p>If <code>average</code> argument is omitted, <code>fmeasure</code> use default value: <code>'-average micro'</code>.</p> +<p>The following query shows the example to obtain F1-score. +Each row value has the same type (<code>int</code> or <code>boolean</code>). +If row value's type is <code>int</code>, <code>1</code> is considered as the positive label, and <code>-1</code> or <code>0</code> is considered as the negative label.</p> +<pre><code class="lang-sql"><span class="hljs-keyword">select</span> fmeasure(truth, predicted, <span class="hljs-string">'-average micro'</span>) <span class="hljs-keyword">from</span> <span class="hljs-keyword">data</span>; +</code></pre> +<blockquote> +<p>0.5</p> +</blockquote> +<p>It should be noted that, since the old <code>f1score(truth, predicted)</code> function simply counts the number of "matched" elements between <code>truth</code> and <code>predicted</code>, the above query is equivalent to:</p> +<pre><code class="lang-sql"><span class="hljs-keyword">select</span> f1score(<span class="hljs-built_in">array</span>(truth), <span class="hljs-built_in">array</span>(predicted)) <span class="hljs-keyword">from</span> <span class="hljs-keyword">data</span>; +</code></pre> +<h3 id="binary-average">Binary average</h3> +<p>If <code>binary</code> is passed to <code>average</code>, <code>True Negative</code> samples are ignored to get F1-score.</p> +<p>The following query shows the example to obtain F1-score with binary average.</p> +<pre><code class="lang-sql"><span class="hljs-keyword">select</span> fmeasure(truth, predicted, <span class="hljs-string">'-average binary'</span>) <span class="hljs-keyword">from</span> <span class="hljs-keyword">data</span>; +</code></pre> +<blockquote> +<p>0.4</p> +</blockquote> +<h2 id="f-measure">F-measure</h2> +<p>F-measure is generalized F1-score and the weighted harmonic mean of recall and precision. +F-measure is computed by the following equation:</p> +<p><span class="katex-display"><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msub><mrow><mi mathvariant="normal">F</mi></mrow><mrow><mi>β</mi></mrow></msub><mo>=</mo><mo>(</mo><mn>1</mn><mo>+</mo><msup><mi>β</mi><mn>2</mn></msup><mo>)</mo><mfrac><mrow><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>∗</mo><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow></mrow><mrow><msup><mi>β</mi><mn>2</mn></msup><mrow><mi mathvariant="normal">p</mi><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="n ormal">i</mi><mi mathvariant="normal">s</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">n</mi></mrow><mo>+</mo><mrow><mi mathvariant="normal">r</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">c</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">l</mi></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex"> +\mathrm{F}_{\beta} = (1+\beta^2) \frac{\mathrm{precision} * \mathrm{r
<TRUNCATED>
