paul-rogers commented on code in PR #13984: URL: https://github.com/apache/druid/pull/13984#discussion_r1169383399
########## examples/quickstart/jupyter-notebooks/docker-jupyter/docker-compose-local.yaml: ########## @@ -0,0 +1,172 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +version: "2.2" + +volumes: + metadata_data: {} + middle_var: {} + historical_var: {} + broker_var: {} + coordinator_var: {} + router_var: {} + druid_shared: {} + + +services: + postgres: Review Comment: Not super-critical, but it would be nice to use the same third-party images here and in the "new" ITs. The ITs use MySQL as its DB. It has setup to create the required user and permissions. See [dependencies.yaml](https://github.com/apache/druid/blob/master/integration-tests-ex/cases/cluster/Common/dependencies.yaml) for the IT setup. ########## docs/tutorials/tutorial-jupyter-docker.md: ########## @@ -0,0 +1,193 @@ +--- +id: tutorial-jupyter-docker +title: "Docker for Jupyter Notebook tutorials" +sidebar_label: "Docker for tutorials" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + + +Apache Druid provides a custom Jupyter container that contains the prerequisites +for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves. +You can run the Jupyter container, as well as containers for Druid and Apache Kafka, +using the Docker Compose file provided in the Druid GitHub repository. + +You can run the following combination of applications: +* [Jupyter only](#start-only-the-jupyter-container) +* [Jupyter and Druid](#start-jupyter-and-druid) +* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka) + +## Prerequisites + +Jupyter in Docker requires that you have **Docker** and **Docker Compose**. +We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/). + +## Launch the Docker containers + +You run Docker Compose to launch Jupyter and optionally Druid or Kafka. +Docker Compose references the configuration in `docker-compose.yaml`. +Running Druid in Docker also requires the `environment` file, which +sets the configuration properties for the Druid services. +To get started, download both `docker-compose.yaml` and `environment` from +[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip). + +Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and +access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`. + +### Start only the Jupyter container + +If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials. +In the same directory as `docker-compose.yaml`, start the application: + +```bash +docker compose --profile jupyter up -d +``` + +The Docker Compose file assigns `8889` for the Jupyter port. +You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application. + +### Start Jupyter and Druid + +Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable. Review Comment: the --> an? We've not talked about an environment file yet so I don't know which one is "the" environment file. ########## docs/tutorials/tutorial-jupyter-docker.md: ########## @@ -0,0 +1,193 @@ +--- +id: tutorial-jupyter-docker +title: "Docker for Jupyter Notebook tutorials" +sidebar_label: "Docker for tutorials" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + + +Apache Druid provides a custom Jupyter container that contains the prerequisites +for all Jupyter-based Druid tutorials, as well as all of the tutorials themselves. +You can run the Jupyter container, as well as containers for Druid and Apache Kafka, +using the Docker Compose file provided in the Druid GitHub repository. + +You can run the following combination of applications: +* [Jupyter only](#start-only-the-jupyter-container) +* [Jupyter and Druid](#start-jupyter-and-druid) +* [Jupyter, Druid, and Kafka](#start-jupyter-druid-and-kafka) + +## Prerequisites + +Jupyter in Docker requires that you have **Docker** and **Docker Compose**. +We recommend installing these through [Docker Desktop](https://docs.docker.com/desktop/). + +## Launch the Docker containers + +You run Docker Compose to launch Jupyter and optionally Druid or Kafka. +Docker Compose references the configuration in `docker-compose.yaml`. +Running Druid in Docker also requires the `environment` file, which +sets the configuration properties for the Druid services. +To get started, download both `docker-compose.yaml` and `environment` from +[`tutorial-jupyter-docker.zip`](https://github.com/apache/druid/blob/master/examples/quickstart/jupyter-notebooks/docker-jupyter/tutorial-jupyter-docker.zip). + +Alternatively, you can clone the [Apache Druid repo](https://github.com/apache/druid) and +access the files in `druid/examples/quickstart/jupyter-notebooks/docker-jupyter`. + +### Start only the Jupyter container + +If you already have Druid running locally, you can run only the Jupyter container to complete the tutorials. +In the same directory as `docker-compose.yaml`, start the application: + +```bash +docker compose --profile jupyter up -d +``` + +The Docker Compose file assigns `8889` for the Jupyter port. +You can override the port number by setting the `JUPYTER_PORT` environment variable before starting the Docker application. + +### Start Jupyter and Druid + +Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable. +`DRUID_VERSION` references the Docker tag for the version of Druid to pull from the +[Apache Druid Docker Hub](https://hub.docker.com/r/apache/druid/tags). + +In the same directory as `docker-compose.yaml` and `environment`, start the application: + +```bash +DRUID_VERSION={{DRUIDVERSION}} docker compose --profile druid-jupyter up -d +``` + +### Start Jupyter, Druid, and Kafka + +Running Druid in Docker requires the `environment` file as well as the `DRUID_VERSION` environment variable. + +In the same directory as `docker-compose.yaml` and `environment`, start the application: + +```bash +DRUID_VERSION={{DRUIDVERSION}} docker compose --profile all-services up -d +``` + +### Update image from Docker Hub + +If you already have a local cache of the Jupyter image, you can update the image before running the application using the following command: Review Comment: When might I want to update the image? Doesn't Docker grab the latest one itself? ########## examples/quickstart/jupyter-notebooks/docker-jupyter/docker-compose.yaml: ########## @@ -0,0 +1,170 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +version: "2.2" + +volumes: + metadata_data: {} + middle_var: {} + historical_var: {} + broker_var: {} + coordinator_var: {} + router_var: {} + druid_shared: {} + + +services: Review Comment: Cool use of the `profiles` feature! There is redundancy between this file and the previous one. Compose allows common services to be defined in a separate file, then imported. See the `dependencies.yaml` mentioned above, along with the corresponding `druid.yaml`. ########## examples/quickstart/jupyter-notebooks/docker-jupyter/environment: ########## @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# Review Comment: Would be great if we could reference the clever encoding scheme used here. Do we explain it elsewhere? If so, perhaps provide a link to newbies can sort out the odd "druid_something" format of the env vars. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
