davidcavazos commented on code in PR #22587:
URL: https://github.com/apache/beam/pull/22587#discussion_r1012106374
##########
examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb:
##########
@@ -0,0 +1,2163 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "source": [
+ "#@title ###### Licensed to the Apache Software Foundation (ASF),
Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n";,
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ],
+ "metadata": {
+ "id": "sARMhsXz8yR1",
+ "cellView": "form"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Overview\n",
+ "\n",
+ "One of the most common tools used for data exploration and
pre-processing is [pandas
DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).
Pandas has become very popular for its ease of use. It has very intuitive
methods to perform common analytical tasks and data pre-processing. \n",
+ "\n",
+ "Pandas loads all of the data into memory on a single machine (one
node) for rapid execution. This works well when dealing with small-scale
datasets. However, many projects involve datasets that can grow too big to fit
in memory. These use cases generally require the usage of parallel data
processing frameworks such as Apache Beam.\n",
+ "\n",
+ "\n",
+ "## Beam DataFrames\n",
+ "\n",
+ "\n",
+ "Beam DataFrames provide a pandas-like\n",
+ "API to declare and define Beam processing pipelines. It provides a
familiar interface for machine learning practioners to build complex
data-processing pipelines by only invoking standard pandas commands.\n",
+ "\n",
+ "> ℹ️ To learn more about Beam DataFrames, take a look at the\n",
+ "[Beam DataFrames
overview](https://beam.apache.org/documentation/dsls/dataframes/overview)
page.\n",
+ "\n",
+ "## Tutorial outline\n",
+ "\n",
+ "In this notebook, we walk through the use of the Beam DataFrames API
to perform common data exploration as well as pre-processing steps that are
necessary to prepare your dataset for machine learning model training and
inference, such as: \n",
+ "\n",
+ "* Removing unwanted columns.\n",
+ "* One-hot encoding categorical columns.\n",
+ "* Normalizing numerical columns.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "iFZC1inKuUCy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Installation\n",
+ "\n",
+ "As we want to explore the elements within a `PCollection`, we can
make use of the the Interactive runner by installing Apache Beam with the
`interactive` component. The latest implemented DataFrames API methods invoked
in this notebook are available in Beam <b>2.41</b> or later.\n"
+ ],
+ "metadata": {
+ "id": "A0f2HJ22D4lt"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pCjwrwNWnuqI"
+ },
+ "source": [
+ "**Option 1:** Install latest version with implemented df.mean()\n",
+ "\n",
+ "TODO: Remove "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "-OJC0Xn5Um-C"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone https://github.com/apache/beam.git\n";,
+ "\n",
+ "!cd beam/sdks/python && pip3 install -r build-requirements.txt \n",
+ "\n",
+ "%pip install -e beam/sdks/python/.[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xfXzNzA1n3ZP"
+ },
+ "source": [
+ "**Option 2:** Install latest release version \n",
+ "\n",
+ "**[12/07/2022]:** df.mean() is currently not supported for this
version (beam 2.40)\n",
+ "\n",
+ "TODO: Remove"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "4xY7ECJZOuJj"
+ },
+ "outputs": [],
+ "source": [
+ "! pip install apache-beam[interactive,gcp]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part I : Local exploration with the Interactive Beam runner\n",
+ "We first use the [Interactive
Beam](https://beam.apache.org/releases/pydoc/2.20.0/apache_beam.runners.interactive.interactive_beam.html)
to explore and develop our pipeline.\n",
+ "This allows us to test our code interactively, building out the
pipeline as we go before deploying it on a distributed runner. \n",
+ "\n",
+ "\n",
+ "> ℹ️ In this section, we will only be working with a subset of the
original dataset since we're only using the the compute resources of the
notebook instance.\n"
+ ],
+ "metadata": {
+ "id": "3NO6RgB7GkkE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5I3G094hoB1P"
+ },
+ "source": [
+ "# Loading the data\n",
+ "\n",
+ "Pandas has the\n",
+
"[`pandas.read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)\n",
+ "function to easily read CSV files into DataFrames.\n",
+ "We're using the beam\n",
+
"[`beam.dataframe.io.read_csv`](https://beam.apache.org/releases/pydoc/current/apache_beam.dataframe.io.html#apache_beam.dataframe.io.read_csv)\n",
+ "function that emulates `pandas.read_csv`. The main difference between
them is that the beam method returns a deferred Beam DataFrame while pandas
return a standard DataFrame.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "id": "X3_OB9cAULav"
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "import numpy as np\n",
+ "import pandas as pd \n",
+ "import apache_beam as beam\n",
+ "import apache_beam.runners.interactive.interactive_beam as ib\n",
+ "from apache_beam import dataframe\n",
+ "from apache_beam.runners.interactive.interactive_runner import
InteractiveRunner\n",
+ "from apache_beam.runners.dataflow import DataflowRunner\n",
+ "\n",
+ "# Available options: [sample_1000, sample_10000, sample_100000, full]
where\n",
+ "# sample contains all of the dataset (around 1000000 samples)\n",
+ "\n",
+ "source_csv_file =
'gs://apache-beam-samples/nasa_jpl_asteroid/sample_10000.csv'\n",
+ "\n",
+ "# Initialize pipline\n",
+ "p = beam.Pipeline(InteractiveRunner())\n",
+ "\n",
+ "# Create a deferred Beam DataFrame with the contents of our csv
file.\n",
+ "beam_df = p | beam.dataframe.io.read_csv(source_csv_file,
splittable=True)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "paf7yf3YpCh8"
+ },
+ "source": [
+ "# Data pre-processing\n",
+ "\n",
+ "## Dataset description \n",
+ "\n",
+ "### [NASA - Nearest Earth Objects
dataset](https://cneos.jpl.nasa.gov/ca/)\n",
+ "There are an innumerable number of objects in the outer space. Some
of them are closer than we think. Even though we might think that a distance of
70,000 Km can not potentially harm us, but at an astronomical scale, this is a
very small distance and can disrupt many natural phenomena. \n",
+ "\n",
+ "These objects/asteroids can thus prove to be harmful. Hence, it is
wise to know what is surrounding us and what can harm us amongst those. Thus,
this dataset compiles the list of NASA certified asteroids that are classified
as the nearest earth object."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "\n",
+ "Let's first inspect the columns of our dataset and their types"
+ ],
+ "metadata": {
+ "id": "cvAu5T0ENjuQ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/";
+ },
+ "id": "LwW77ixE-pjR",
+ "outputId": "f5386993-14cb-42ee-94ca-8ea006860d3e"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "spk_id int64\n",
+ "full_name object\n",
+ "near_earth_object object\n",
+ "absolute_magnitude float64\n",
+ "diameter float64\n",
+ "albedo float64\n",
+ "diameter_sigma float64\n",
+ "eccentricity float64\n",
+ "inclination float64\n",
+ "moid_ld float64\n",
+ "object_class object\n",
+ "semi_major_axis_au_unit float64\n",
+ "hazardous_flag object\n",
+ "dtype: object"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 3
+ }
+ ],
+ "source": [
+ "beam_df.dtypes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "When using Interactive Beam, we can use `ib.collect()` to bring a
Beam DataFrame into local memory as a Pandas DataFrame."
+ ],
+ "metadata": {
+ "id": "1Wa6fpbyQige"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/";,
+ "height": 378
+ },
+ "id": "DPxkAmkpq4Xv",
+ "outputId": "e49b4243-107f-4256-9e09-49cc20bf7f56"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\"
href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\";
integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\"
crossorigin=\"anonymous\">\n",
+ " <div
id=\"progress_indicator_f6af66571c53daa0d9052370b7d1d8b7\">\n",
+ " <div class=\"spinner-border text-info\"
role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing...
collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery ==
'undefined') {\n",
+ " var jqueryScript =
document.createElement('script');\n",
+ " jqueryScript.src =
'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript =
document.createElement('script');\n",
+ " datatableScript.src =
'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery =
jQuery.noConflict(true);\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_f6af66571c53daa0d9052370b7d1d8b7\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_f6af66571c53daa0d9052370b7d1d8b7\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " spk_id absolute_magnitude diameter
albedo \\\n",
+ "count 9.999000e+03 9999.000000 8688.000000
8672.000000 \n",
+ "mean 2.005000e+06 12.675380 19.245446
0.197723 \n",
+ "std 2.886607e+03 1.639609 30.190191
0.138819 \n",
+ "min 2.000001e+06 3.000000 0.300000
0.008000 \n",
+ "25% 2.002500e+06 11.900000 5.614000
0.074000 \n",
+ "50% 2.005000e+06 12.900000 9.814000
0.187000 \n",
+ "75% 2.007500e+06 13.700000 19.156750
0.283000 \n",
+ "max 2.009999e+06 20.700000 939.400000
1.000000 \n",
+ "\n",
+ " diameter_sigma eccentricity inclination moid_ld
\\\n",
+ "count 8591.000000 9999.000000 9999.000000 9999.000000
\n",
+ "mean 0.454072 0.148716 7.890742 509.805237
\n",
+ "std 1.093676 0.083803 6.336244 205.046582
\n",
+ "min 0.006000 0.001003 0.042716 0.131028
\n",
+ "25% 0.120000 0.093780 3.220137 377.829197
\n",
+ "50% 0.201000 0.140335 6.018836 470.650523
\n",
+ "75% 0.375000 0.187092 10.918176 636.010802
\n",
+ "max 39.297000 0.889831 68.018875 4241.524913
\n",
+ "\n",
+ " semi_major_axis_au_unit \n",
+ "count 9999.000000 \n",
+ "mean 2.689836 \n",
+ "std 0.607190 \n",
+ "min 0.832048 \n",
+ "25% 2.340816 \n",
+ "50% 2.614468 \n",
+ "75% 3.005449 \n",
+ "max 24.667968 "
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-2c0349a9-81c4-473a-9fa1-44c423244858\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>spk_id</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>diameter</th>\n",
+ " <th>albedo</th>\n",
+ " <th>diameter_sigma</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>count</th>\n",
+ " <td>9.999000e+03</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>8688.000000</td>\n",
+ " <td>8672.000000</td>\n",
+ " <td>8591.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>mean</th>\n",
+ " <td>2.005000e+06</td>\n",
+ " <td>12.675380</td>\n",
+ " <td>19.245446</td>\n",
+ " <td>0.197723</td>\n",
+ " <td>0.454072</td>\n",
+ " <td>0.148716</td>\n",
+ " <td>7.890742</td>\n",
+ " <td>509.805237</td>\n",
+ " <td>2.689836</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>std</th>\n",
+ " <td>2.886607e+03</td>\n",
+ " <td>1.639609</td>\n",
+ " <td>30.190191</td>\n",
+ " <td>0.138819</td>\n",
+ " <td>1.093676</td>\n",
+ " <td>0.083803</td>\n",
+ " <td>6.336244</td>\n",
+ " <td>205.046582</td>\n",
+ " <td>0.607190</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>min</th>\n",
+ " <td>2.000001e+06</td>\n",
+ " <td>3.000000</td>\n",
+ " <td>0.300000</td>\n",
+ " <td>0.008000</td>\n",
+ " <td>0.006000</td>\n",
+ " <td>0.001003</td>\n",
+ " <td>0.042716</td>\n",
+ " <td>0.131028</td>\n",
+ " <td>0.832048</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>25%</th>\n",
+ " <td>2.002500e+06</td>\n",
+ " <td>11.900000</td>\n",
+ " <td>5.614000</td>\n",
+ " <td>0.074000</td>\n",
+ " <td>0.120000</td>\n",
+ " <td>0.093780</td>\n",
+ " <td>3.220137</td>\n",
+ " <td>377.829197</td>\n",
+ " <td>2.340816</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>50%</th>\n",
+ " <td>2.005000e+06</td>\n",
+ " <td>12.900000</td>\n",
+ " <td>9.814000</td>\n",
+ " <td>0.187000</td>\n",
+ " <td>0.201000</td>\n",
+ " <td>0.140335</td>\n",
+ " <td>6.018836</td>\n",
+ " <td>470.650523</td>\n",
+ " <td>2.614468</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>75%</th>\n",
+ " <td>2.007500e+06</td>\n",
+ " <td>13.700000</td>\n",
+ " <td>19.156750</td>\n",
+ " <td>0.283000</td>\n",
+ " <td>0.375000</td>\n",
+ " <td>0.187092</td>\n",
+ " <td>10.918176</td>\n",
+ " <td>636.010802</td>\n",
+ " <td>3.005449</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>max</th>\n",
+ " <td>2.009999e+06</td>\n",
+ " <td>20.700000</td>\n",
+ " <td>939.400000</td>\n",
+ " <td>1.000000</td>\n",
+ " <td>39.297000</td>\n",
+ " <td>0.889831</td>\n",
+ " <td>68.018875</td>\n",
+ " <td>4241.524913</td>\n",
+ " <td>24.667968</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\"
onclick=\"convertToInteractive('df-2c0349a9-81c4-473a-9fa1-44c423244858')\"\n",
+ " title=\"Convert this dataframe to an interactive
table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\";
height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06
2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41
7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72
7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2
1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72
1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px
3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ "
document.querySelector('#df-2c0349a9-81c4-473a-9fa1-44c423244858
button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' :
'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element =
document.querySelector('#df-2c0349a9-81c4-473a-9fa1-44c423244858');\n",
+ " const dataTable =\n",
+ " await
google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key],
{});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the '
+\n",
+ " '<a target=\"_blank\"
href=https://colab.research.google.com/notebooks/data_table.ipynb>data table
notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable,
element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 8
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can see that our datasets consists of both:\n",
+ "\n",
+ "* **Numerical columns:** These columns need to be transformed through
[normalization](https://developers.google.com/machine-learning/data-prep/transform/normalization)
before they can be used for training a machine learning model.\n",
+ "\n",
+ "* **Categorical columns:** We need to transform those columns with
[one-hot
encoding](https://developers.google.com/machine-learning/data-prep/transform/transform-categorical)
to use them during training. \n"
+ ],
+ "metadata": {
+ "id": "8jV9odKhNyF2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can also explore use the standard pandas command
`DataFrame.describe()` to generate descriptive statistics for the numerical
columns like percentile, mean, std etc. "
+ ],
+ "metadata": {
+ "id": "MGAErO0lAYws"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "with dataframe.allow_non_parallel_operations():\n",
+ " beam_df_description = ib.collect(beam_df.describe())\n",
+ "\n",
+ "beam_df_description"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/";,
+ "height": 378
+ },
+ "id": "Befv697VBGM7",
+ "outputId": "d02b7a41-a8a3-4837-cf63-e1fa9e7b011e"
+ },
+ "execution_count": 9,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\"
href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\";
integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\"
crossorigin=\"anonymous\">\n",
+ " <div
id=\"progress_indicator_a5b31481d153dff1b7ecdd673624949b\">\n",
+ " <div class=\"spinner-border text-info\"
role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing...
collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery ==
'undefined') {\n",
+ " var jqueryScript =
document.createElement('script');\n",
+ " jqueryScript.src =
'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript =
document.createElement('script');\n",
+ " datatableScript.src =
'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery =
jQuery.noConflict(true);\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_a5b31481d153dff1b7ecdd673624949b\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_a5b31481d153dff1b7ecdd673624949b\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " spk_id absolute_magnitude diameter
albedo \\\n",
+ "count 9.999000e+03 9999.000000 8688.000000
8672.000000 \n",
+ "mean 2.005000e+06 12.675380 19.245446
0.197723 \n",
+ "std 2.886607e+03 1.639609 30.190191
0.138819 \n",
+ "min 2.000001e+06 3.000000 0.300000
0.008000 \n",
+ "25% 2.002500e+06 11.900000 5.614000
0.074000 \n",
+ "50% 2.005000e+06 12.900000 9.814000
0.187000 \n",
+ "75% 2.007500e+06 13.700000 19.156750
0.283000 \n",
+ "max 2.009999e+06 20.700000 939.400000
1.000000 \n",
+ "\n",
+ " diameter_sigma eccentricity inclination moid_ld
\\\n",
+ "count 8591.000000 9999.000000 9999.000000 9999.000000
\n",
+ "mean 0.454072 0.148716 7.890742 509.805237
\n",
+ "std 1.093676 0.083803 6.336244 205.046582
\n",
+ "min 0.006000 0.001003 0.042716 0.131028
\n",
+ "25% 0.120000 0.093780 3.220137 377.829197
\n",
+ "50% 0.201000 0.140335 6.018836 470.650523
\n",
+ "75% 0.375000 0.187092 10.918176 636.010802
\n",
+ "max 39.297000 0.889831 68.018875 4241.524913
\n",
+ "\n",
+ " semi_major_axis_au_unit \n",
+ "count 9999.000000 \n",
+ "mean 2.689836 \n",
+ "std 0.607190 \n",
+ "min 0.832048 \n",
+ "25% 2.340816 \n",
+ "50% 2.614468 \n",
+ "75% 3.005449 \n",
+ "max 24.667968 "
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-d16cf806-a3e2-46d9-973d-74448570aaa2\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>spk_id</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>diameter</th>\n",
+ " <th>albedo</th>\n",
+ " <th>diameter_sigma</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>count</th>\n",
+ " <td>9.999000e+03</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>8688.000000</td>\n",
+ " <td>8672.000000</td>\n",
+ " <td>8591.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " <td>9999.000000</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>mean</th>\n",
+ " <td>2.005000e+06</td>\n",
+ " <td>12.675380</td>\n",
+ " <td>19.245446</td>\n",
+ " <td>0.197723</td>\n",
+ " <td>0.454072</td>\n",
+ " <td>0.148716</td>\n",
+ " <td>7.890742</td>\n",
+ " <td>509.805237</td>\n",
+ " <td>2.689836</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>std</th>\n",
+ " <td>2.886607e+03</td>\n",
+ " <td>1.639609</td>\n",
+ " <td>30.190191</td>\n",
+ " <td>0.138819</td>\n",
+ " <td>1.093676</td>\n",
+ " <td>0.083803</td>\n",
+ " <td>6.336244</td>\n",
+ " <td>205.046582</td>\n",
+ " <td>0.607190</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>min</th>\n",
+ " <td>2.000001e+06</td>\n",
+ " <td>3.000000</td>\n",
+ " <td>0.300000</td>\n",
+ " <td>0.008000</td>\n",
+ " <td>0.006000</td>\n",
+ " <td>0.001003</td>\n",
+ " <td>0.042716</td>\n",
+ " <td>0.131028</td>\n",
+ " <td>0.832048</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>25%</th>\n",
+ " <td>2.002500e+06</td>\n",
+ " <td>11.900000</td>\n",
+ " <td>5.614000</td>\n",
+ " <td>0.074000</td>\n",
+ " <td>0.120000</td>\n",
+ " <td>0.093780</td>\n",
+ " <td>3.220137</td>\n",
+ " <td>377.829197</td>\n",
+ " <td>2.340816</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>50%</th>\n",
+ " <td>2.005000e+06</td>\n",
+ " <td>12.900000</td>\n",
+ " <td>9.814000</td>\n",
+ " <td>0.187000</td>\n",
+ " <td>0.201000</td>\n",
+ " <td>0.140335</td>\n",
+ " <td>6.018836</td>\n",
+ " <td>470.650523</td>\n",
+ " <td>2.614468</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>75%</th>\n",
+ " <td>2.007500e+06</td>\n",
+ " <td>13.700000</td>\n",
+ " <td>19.156750</td>\n",
+ " <td>0.283000</td>\n",
+ " <td>0.375000</td>\n",
+ " <td>0.187092</td>\n",
+ " <td>10.918176</td>\n",
+ " <td>636.010802</td>\n",
+ " <td>3.005449</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>max</th>\n",
+ " <td>2.009999e+06</td>\n",
+ " <td>20.700000</td>\n",
+ " <td>939.400000</td>\n",
+ " <td>1.000000</td>\n",
+ " <td>39.297000</td>\n",
+ " <td>0.889831</td>\n",
+ " <td>68.018875</td>\n",
+ " <td>4241.524913</td>\n",
+ " <td>24.667968</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\"
onclick=\"convertToInteractive('df-d16cf806-a3e2-46d9-973d-74448570aaa2')\"\n",
+ " title=\"Convert this dataframe to an interactive
table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\";
height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06
2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41
7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72
7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2
1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72
1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px
3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ "
document.querySelector('#df-d16cf806-a3e2-46d9-973d-74448570aaa2
button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' :
'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element =
document.querySelector('#df-d16cf806-a3e2-46d9-973d-74448570aaa2');\n",
+ " const dataTable =\n",
+ " await
google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key],
{});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the '
+\n",
+ " '<a target=\"_blank\"
href=https://colab.research.google.com/notebooks/data_table.ipynb>data table
notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable,
element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 9
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "D9uJtHLSSAMC"
+ },
+ "source": [
+ "Before executing any transformations, we need to check if all the
columns need to be used for model training. Let's first have a look at the
column description as provided by the [JPL
website](https://ssd.jpl.nasa.gov/sbdb_query.cgi):\n",
+ "\n",
+ "* **spk_id:** Object primary SPK-ID\n",
+ "* **full_name:** Asteroid name\n",
+ "* **near_earth_object:** Near-earth object flag\n",
+ "* **absolute_magnitude:** the apparent magnitude an object would have
if it were located at a distance of 10 parsecs.\n",
+ "* **diameter:** object diameter (from equivalent sphere) km Unit\n",
+ "* **albedo:** a measure of the diffuse reflection of solar radiation
out of the total solar radiation and measured on a scale from 0 to 1.\n",
+ "* **diameter_sigma:** 1-sigma uncertainty in object diameter km
Unit.\n",
+ "* **eccentricity:** value between 0 and 1 that referes to how flat or
round the shape of the asteroid is \n",
+ "* **inclination:** angle with respect to x-y ecliptic plane\n",
+ "* **moid_ld:** Earth Minimum Orbit Intersection Distance au Unit\n",
+ "* **object_class:** the classification of the asteroid. Checkout this
[link](https://pdssbn.astro.umd.edu/data_other/objclass.shtml) for a more
detailed description.\n",
+ "* **Semi-major axis au Unit:** the length of half of the long axis in
AU unit\n",
+ "* **hazardous_flag:** Hazardous Asteroid Flag"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "DzYVKbwTp72d"
+ },
+ "source": [
+ "Columns **'spkid'** and **'full_name'** are unique for each row.
These columns can be removed since they are not needed for model training."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "piRPwH2aqT06"
+ },
+ "outputs": [],
+ "source": [
+ "beam_df = beam_df.drop(['spk_id', 'full_name'], axis='columns',
inplace=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fRvNyahSuX_y"
+ },
+ "source": [
+ "Let's have a look at the number of missing values"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/";,
+ "height": 353
+ },
+ "id": "A2PLchW8vXvt",
+ "outputId": "c08d7f23-3a48-4282-a252-66f73cc7fd86"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+
"/content/beam/sdks/python/apache_beam/dataframe/frame_base.py:145:
RuntimeWarning: invalid value encountered in long_scalars\n",
+ " lambda left, right: getattr(left, op)(right), name=op,
args=[other])\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\"
href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\";
integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\"
crossorigin=\"anonymous\">\n",
+ " <div
id=\"progress_indicator_02cba1067de8e024374a26297834d233\">\n",
+ " <div class=\"spinner-border text-info\"
role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing...
collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery ==
'undefined') {\n",
+ " var jqueryScript =
document.createElement('script');\n",
+ " jqueryScript.src =
'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript =
document.createElement('script');\n",
+ " datatableScript.src =
'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery =
jQuery.noConflict(true);\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_02cba1067de8e024374a26297834d233\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "near_earth_object 0.000000\n",
+ "absolute_magnitude 0.000000\n",
+ "diameter 13.111311\n",
+ "albedo 13.271327\n",
+ "diameter_sigma 14.081408\n",
+ "eccentricity 0.000000\n",
+ "inclination 0.000000\n",
+ "moid_ld 0.000000\n",
+ "object_class 0.000000\n",
+ "semi_major_axis_au_unit 0.000000\n",
+ "hazardous_flag 0.000000\n",
+ "dtype: float64"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 7
+ }
+ ],
+ "source": [
+ "ib.collect(beam_df.isnull().mean() * 100)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "00MRdFGLwQiD"
+ },
+ "source": [
+ "It can be observed that most of the columns do not have missing
values. However, columns **'diameter'**, **'albedo'** and **'diameter_sigma'**
have many missing values. Since these values cannot be measured or derived, we
can remove them since they will not be required for training the machine
learning model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "tHYeCHREwvyB",
+ "colab": {
+ "base_uri": "https://localhost:8080/";,
+ "height": 538
+ },
+ "outputId": "5b1b2767-6ae9-4920-f96e-fd1f18e697bb"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ],
+ "text/html": [
+ "\n",
+ " <link rel=\"stylesheet\"
href=\"https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css\";
integrity=\"sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh\"
crossorigin=\"anonymous\">\n",
+ " <div
id=\"progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\">\n",
+ " <div class=\"spinner-border text-info\"
role=\"status\"></div>\n",
+ " <span class=\"text-info\">Processing...
collect</span>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/javascript": [
+ "\n",
+ " if (typeof window.interactive_beam_jquery ==
'undefined') {\n",
+ " var jqueryScript =
document.createElement('script');\n",
+ " jqueryScript.src =
'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
+ " jqueryScript.type = 'text/javascript';\n",
+ " jqueryScript.onload = function() {\n",
+ " var datatableScript =
document.createElement('script');\n",
+ " datatableScript.src =
'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
+ " datatableScript.type = 'text/javascript';\n",
+ " datatableScript.onload = function() {\n",
+ " window.interactive_beam_jquery =
jQuery.noConflict(true);\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }\n",
+ " document.head.appendChild(datatableScript);\n",
+ " };\n",
+ " document.head.appendChild(jqueryScript);\n",
+ " } else {\n",
+ "
window.interactive_beam_jquery(document).ready(function($){\n",
+ " \n",
+ "
$(\"#progress_indicator_0a93a40e87f23e4a235dfd56cd10188b\").remove();\n",
+ " });\n",
+ " }"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " near_earth_object absolute_magnitude eccentricity
inclination \\\n",
+ "0 N 3.40 0.076009
10.594067 \n",
+ "1 N 4.20 0.229972
34.832932 \n",
+ "2 N 5.33 0.256936
12.991043 \n",
+ "3 N 3.00 0.088721
7.141771 \n",
+ "4 N 6.90 0.190913
5.367427 \n",
+ "... ... ... ...
... \n",
+ "9994 N 15.10 0.160610
2.311731 \n",
+ "9995 N 13.60 0.235174
7.657713 \n",
+ "9996 N 14.30 0.113059
2.459643 \n",
+ "9997 N 15.10 0.093852
3.912263 \n",
+ "9998 N 13.00 0.071351
3.198839 \n",
+ "\n",
+ " moid_ld object_class semi_major_axis_au_unit
hazardous_flag \n",
+ "0 620.640533 MBA 2.769165
N \n",
+ "1 480.348639 MBA 2.773841
N \n",
+ "2 402.514639 MBA 2.668285
N \n",
+ "3 443.451432 MBA 2.361418
N \n",
+ "4 426.433027 MBA 2.574037
N \n",
+ "... ... ... ...
... \n",
+ "9994 388.723233 MBA 2.390249
N \n",
+ "9995 444.194746 MBA 2.796605
N \n",
+ "9996 495.460110 MBA 2.545674
N \n",
+ "9997 373.848377 MBA 2.160961
N \n",
+ "9998 632.144398 MBA 2.839917
N \n",
+ "\n",
+ "[9999 rows x 8 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " <div id=\"df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398\">\n",
+ " <div class=\"colab-df-container\">\n",
+ " <div>\n",
+ "<style scoped>\n",
+ " .dataframe tbody tr th:only-of-type {\n",
+ " vertical-align: middle;\n",
+ " }\n",
+ "\n",
+ " .dataframe tbody tr th {\n",
+ " vertical-align: top;\n",
+ " }\n",
+ "\n",
+ " .dataframe thead th {\n",
+ " text-align: right;\n",
+ " }\n",
+ "</style>\n",
+ "<table border=\"1\" class=\"dataframe\">\n",
+ " <thead>\n",
+ " <tr style=\"text-align: right;\">\n",
+ " <th></th>\n",
+ " <th>near_earth_object</th>\n",
+ " <th>absolute_magnitude</th>\n",
+ " <th>eccentricity</th>\n",
+ " <th>inclination</th>\n",
+ " <th>moid_ld</th>\n",
+ " <th>object_class</th>\n",
+ " <th>semi_major_axis_au_unit</th>\n",
+ " <th>hazardous_flag</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th>0</th>\n",
+ " <td>N</td>\n",
+ " <td>3.40</td>\n",
+ " <td>0.076009</td>\n",
+ " <td>10.594067</td>\n",
+ " <td>620.640533</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.769165</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>1</th>\n",
+ " <td>N</td>\n",
+ " <td>4.20</td>\n",
+ " <td>0.229972</td>\n",
+ " <td>34.832932</td>\n",
+ " <td>480.348639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.773841</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>2</th>\n",
+ " <td>N</td>\n",
+ " <td>5.33</td>\n",
+ " <td>0.256936</td>\n",
+ " <td>12.991043</td>\n",
+ " <td>402.514639</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.668285</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>3</th>\n",
+ " <td>N</td>\n",
+ " <td>3.00</td>\n",
+ " <td>0.088721</td>\n",
+ " <td>7.141771</td>\n",
+ " <td>443.451432</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.361418</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>4</th>\n",
+ " <td>N</td>\n",
+ " <td>6.90</td>\n",
+ " <td>0.190913</td>\n",
+ " <td>5.367427</td>\n",
+ " <td>426.433027</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.574037</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>...</th>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " <td>...</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9994</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.160610</td>\n",
+ " <td>2.311731</td>\n",
+ " <td>388.723233</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.390249</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9995</th>\n",
+ " <td>N</td>\n",
+ " <td>13.60</td>\n",
+ " <td>0.235174</td>\n",
+ " <td>7.657713</td>\n",
+ " <td>444.194746</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.796605</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9996</th>\n",
+ " <td>N</td>\n",
+ " <td>14.30</td>\n",
+ " <td>0.113059</td>\n",
+ " <td>2.459643</td>\n",
+ " <td>495.460110</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.545674</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9997</th>\n",
+ " <td>N</td>\n",
+ " <td>15.10</td>\n",
+ " <td>0.093852</td>\n",
+ " <td>3.912263</td>\n",
+ " <td>373.848377</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.160961</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th>9998</th>\n",
+ " <td>N</td>\n",
+ " <td>13.00</td>\n",
+ " <td>0.071351</td>\n",
+ " <td>3.198839</td>\n",
+ " <td>632.144398</td>\n",
+ " <td>MBA</td>\n",
+ " <td>2.839917</td>\n",
+ " <td>N</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n",
+ "<p>9999 rows × 8 columns</p>\n",
+ "</div>\n",
+ " <button class=\"colab-df-convert\"
onclick=\"convertToInteractive('df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398')\"\n",
+ " title=\"Convert this dataframe to an interactive
table.\"\n",
+ " style=\"display:none;\">\n",
+ " \n",
+ " <svg xmlns=\"http://www.w3.org/2000/svg\";
height=\"24px\"viewBox=\"0 0 24 24\"\n",
+ " width=\"24px\">\n",
+ " <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
+ " <path d=\"M18.56 5.44l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06
2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06
2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41
7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72
7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2
1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72
1.47 1.35L5.41 20z\"/>\n",
+ " </svg>\n",
+ " </button>\n",
+ " \n",
+ " <style>\n",
+ " .colab-df-container {\n",
+ " display:flex;\n",
+ " flex-wrap:wrap;\n",
+ " gap: 12px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert {\n",
+ " background-color: #E8F0FE;\n",
+ " border: none;\n",
+ " border-radius: 50%;\n",
+ " cursor: pointer;\n",
+ " display: none;\n",
+ " fill: #1967D2;\n",
+ " height: 32px;\n",
+ " padding: 0 0 0 0;\n",
+ " width: 32px;\n",
+ " }\n",
+ "\n",
+ " .colab-df-convert:hover {\n",
+ " background-color: #E2EBFA;\n",
+ " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px
3px 1px rgba(60, 64, 67, 0.15);\n",
+ " fill: #174EA6;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert {\n",
+ " background-color: #3B4455;\n",
+ " fill: #D2E3FC;\n",
+ " }\n",
+ "\n",
+ " [theme=dark] .colab-df-convert:hover {\n",
+ " background-color: #434B5C;\n",
+ " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
+ " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
+ " fill: #FFFFFF;\n",
+ " }\n",
+ " </style>\n",
+ "\n",
+ " <script>\n",
+ " const buttonEl =\n",
+ "
document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398
button.colab-df-convert');\n",
+ " buttonEl.style.display =\n",
+ " google.colab.kernel.accessAllowed ? 'block' :
'none';\n",
+ "\n",
+ " async function convertToInteractive(key) {\n",
+ " const element =
document.querySelector('#df-f1eb18dc-39ee-4c7c-9c07-b3f2e37fd398');\n",
+ " const dataTable =\n",
+ " await
google.colab.kernel.invokeFunction('convertToInteractive',\n",
+ " [key],
{});\n",
+ " if (!dataTable) return;\n",
+ "\n",
+ " const docLinkHtml = 'Like what you see? Visit the '
+\n",
+ " '<a target=\"_blank\"
href=https://colab.research.google.com/notebooks/data_table.ipynb>data table
notebook</a>'\n",
+ " + ' to learn more about interactive tables.';\n",
+ " element.innerHTML = '';\n",
+ " dataTable['output_type'] = 'display_data';\n",
+ " await google.colab.output.renderOutput(dataTable,
element);\n",
+ " const docLink = document.createElement('div');\n",
+ " docLink.innerHTML = docLinkHtml;\n",
+ " element.appendChild(docLink);\n",
+ " }\n",
+ " </script>\n",
+ " </div>\n",
+ " </div>\n",
+ " "
+ ]
+ },
+ "metadata": {},
+ "execution_count": 8
+ }
+ ],
+ "source": [
+ "beam_df = beam_df.drop(['diameter', 'albedo', 'diameter_sigma'],
axis='columns', inplace=False)\n",
+ "ib.collect(beam_df)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "a3PojL3WBqgE"
+ },
+ "source": [
+ "Next, we need to normalize the numerical columns before using them to
train a model. A common method of standarization is to subtract the mean and
divide by standard deviation (a.k.a
[z-score](https://developers.google.com/machine-learning/data-prep/transform/normalization#z-score)).
This improves the performance and trainign stability of the model during
training and inference.\n"
Review Comment:
Typo: trainign -> training (still present)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
