liferoad commented on code in PR #34845:
URL: https://github.com/apache/beam/pull/34845#discussion_r2074098808
##########
examples/notebooks/beam-ml/anomaly_detection/anomaly_detection_iforest.ipynb:
##########
@@ -0,0 +1,477 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "provenance": [],
+ "authorship_tag": "ABX9TyMFxdjrFa6MkZv9RHCO0MZM"
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "cells": [
+ {
+ "cell_type": "code",
+ "source": [
+ "# @title ###### Licensed to the Apache Software Foundation (ASF),
Version 2.0 (the \"License\")\n",
+ "\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n";,
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License"
+ ],
+ "metadata": {
+ "cellView": "form",
+ "id": "IA9uYREbI3m3"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Use Apache Beam for Anomaly Detection on Streaming Data (Isolation
Forest)\n",
+ "\n",
+ "This notebook demonstrates how to perform anomaly detection on
streaming data using the AnomalyDetection PTransform, a new feature introduced
in Apache Beam 2.64.0 with more improvement on offline model support in
2.65.0.\n",
+ "\n",
+ "We will first fetch the data set of Statlog (shuttle) from UCI
Machine Learning Repository (cached in
gs://apache-beam-samples/anomaly_detection/shuttle/, original link:
https://archive.ics.uci.edu/dataset/148/statlog+shuttle). This data will be
streamed into the pipeline following a periodic impulse. Our Beam pipeline will
then apply the AnomalyDetection PTransform with the a pre-trained isolation
forest model, compute model metrics.\n",
+ "\n",
+ "We demonstrate running the pipeline with our new local runner Prism."
+ ],
+ "metadata": {
+ "id": "Eef1VqflIrXW"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Setting Environment Variables"
+ ],
+ "metadata": {
+ "id": "EKkFQE8iwT7M"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "L_18eUf7QU2I"
+ },
+ "outputs": [],
+ "source": [
+ "# GCP project id\n",
+ "PROJECT_ID = 'apache-beam-testing' # @param {type:'string'}\n",
+ "\n",
+ "# TODO: Change this to an official release once 2.65.0 is
available\n",
+ "BEAM_VERSION = '2.65.0rc1'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from google.colab import auth\n",
+ "auth.authenticate_user(project_id=PROJECT_ID)"
+ ],
+ "metadata": {
+ "id": "A_49Y2aTQeiH"
+ },
+ "execution_count": 2,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Installing Beam and Other Dependencies"
+ ],
+ "metadata": {
+ "id": "0WXpo4aDwG3N"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# For running with local prism runner\n",
+ "!pip install 'apache_beam[interactive]=={BEAM_VERSION}' --quiet\n",
+ "\n",
+ "# # For running with dataflow runner\n",
+ "# !pip install 'apache_beam[gcp, interactive]=={BEAM_VERSION}'
--quiet"
+ ],
+ "metadata": {
+ "id": "5hpDAMOyQfHP"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Download the latest prism\n",
+ "# TODO: We don't need this step once 2.65.0 is available.\n",
+ "! wget
https://dist.apache.org/repos/dist/dev/beam/2.65.0/prism/linux/amd64/apache_beam-v2.65.0-prism-linux-amd64.zip";
+ ],
+ "metadata": {
+ "id": "jAKBvrgq-J3f"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Install pyod for offline anomaly detectors\n",
+ "!pip install pyod==2.0.3"
+ ],
+ "metadata": {
+ "id": "KlkX-iwVm42J"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Training an Offline Isolation Forest Model"
+ ],
+ "metadata": {
+ "id": "UIzbsGtWto5X"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Download the sample data from GCS\n",
+ "train_data_fn = \"./shuttle.trn\"\n",
+ "! gcloud storage cp
\"gs://apache-beam-samples/anomaly_detection/shuttle/shuttle.trn\"
{train_data_fn}"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/";
+ },
+ "id": "vb6ubiSyipuG",
+ "outputId": "e6b0c667-b6f8-43a3-ffd5-344c33ba02e1"
+ },
+ "execution_count": 6,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Copying
gs://apache-beam-samples/anomaly_detection/shuttle/shuttle.trn to
file://./shuttle.trn\n",
+ "\n",
+ "Average throughput: 180.7MiB/s\n"
Review Comment:
clear all cells?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
