paleolimbot commented on code in PR #238:
URL: https://github.com/apache/sedona-db/pull/238#discussion_r2462330613
##########
docs/delta-lake.ipynb:
##########
@@ -65,13 +65,11 @@
"outputs": [],
"source": [
"countries.to_view(\"countries\")\n",
- "df = sd.sql(\"select name, continent, ST_AsText(geometry) as geometry_wkt
from countries\")\n",
+ "df = sd.sql(\n",
+ " \"select name, continent, ST_AsText(geometry) as geometry_wkt from
countries\"\n",
+ ")\n",
"table_path = \"/tmp/delta_with_wkt\"\n",
- "write_deltalake(\n",
- " table_path,\n",
- " df.to_pandas(),\n",
- " mode=\"overwrite\"\n",
- ")"
+ "write_deltalake(table_path, df.to_pandas(), mode=\"overwrite\")"
Review Comment:
It's worth trying this:
```python
sd.sql("SET datafusion.execution.parquet.schema_force_view_types =
false").execute()
```
...and using the `df` object directly. You really want to be streaming the
query result into DeltaLake so that you're not limited to the size of the
memory.
##########
docs/delta-lake.ipynb:
##########
@@ -0,0 +1,242 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "3425268d-6430-4e52-9019-969d61ef5458",
+ "metadata": {},
+ "source": [
+ "# SedonaDB + Delta Lake\n",
+ "\n",
+ "This page shows how to read and write Delta Lake tables with SedonaDB.\n",
+ "\n",
+ "Make sure you run `pip install deltalake` to run the cells in this
notebook."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "83f5ca25-8059-4624-bd00-e44cd172d9c2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from deltalake import write_deltalake, DeltaTable\n",
+ "import sedona.db\n",
+ "\n",
+ "sd = sedona.db.connect()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1a90b1bf-99f9-4f50-987b-f1f30bf9988a",
+ "metadata": {},
+ "source": [
+ "Read in a GeoParquet dataset into a SedonaDB DataFrame."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "a37f3a30-3267-4bfc-8e5b-e234053927af",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "countries = sd.read_parquet(\n",
+ "
\"https://raw.githubusercontent.com/geoarrow/geoarrow-data/v0.2.0/natural-earth/files/natural-earth_countries_geo.parquet\"\n";,
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1f82b047-8182-4120-82d7-945ff38ecbca",
+ "metadata": {},
+ "source": [
+ "## Create a Delta Lake table\n",
+ "\n",
+ "Now write the DataFrame to a Delta Lake table. Notice that the geometry
column must be converted to Well-Known Text (WKT) before writing to the Delta
table.\n",
+ "\n",
+ "Delta Lake does not support geometry columns."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "35bdc296-d9ef-4ea2-9ba5-dac1cfd115ef",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "countries.to_view(\"countries\")\n",
+ "df = sd.sql(\n",
+ " \"select name, continent, ST_AsText(geometry) as geometry_wkt from
countries\"\n",
+ ")\n",
+ "table_path = \"/tmp/delta_with_wkt\"\n",
+ "write_deltalake(table_path, df.to_pandas(), mode=\"overwrite\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "974a5558-fb18-49b7-a304-fa93bdd363e8",
+ "metadata": {},
+ "source": [
+ "## Read Delta table into SedonaDB\n",
+ "\n",
+ "Now read the Delta table back into a SedonaDB DataFrame."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "c15d4605-483a-4041-973a-547098bdeef4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+
"┌─────────────────────────────┬───────────────┬────────────────────────────────────────────────────┐\n",
+ "│ name ┆ continent ┆
geometry_wkt │\n",
+ "│ utf8 ┆ utf8 ┆
utf8 │\n",
+
"╞═════════════════════════════╪═══════════════╪════════════════════════════════════════════════════╡\n",
+ "│ Fiji ┆ Oceania ┆ MULTIPOLYGON(((180
-16.067132663642447,180 -16.55… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ United Republic of Tanzania ┆ Africa ┆
POLYGON((33.90371119710453 -0.9500000000000001,34… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Western Sahara ┆ Africa ┆
POLYGON((-8.665589565454809 27.656425889592356,-8… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Canada ┆ North America ┆
MULTIPOLYGON(((-122.84000000000003 49.00000000000… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ United States of America ┆ North America ┆
MULTIPOLYGON(((-122.84000000000003 49.00000000000… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Kazakhstan ┆ Asia ┆
POLYGON((87.35997033076265 49.21498078062912,86.5… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Uzbekistan ┆ Asia ┆
POLYGON((55.96819135928291 41.30864166926936,55.9… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Papua New Guinea ┆ Oceania ┆
MULTIPOLYGON(((141.00021040259185 -2.600151055515… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Indonesia ┆ Asia ┆
MULTIPOLYGON(((141.00021040259185 -2.600151055515… │\n",
+
"├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤\n",
+ "│ Argentina ┆ South America ┆
MULTIPOLYGON(((-68.63401022758323 -52.63637045887… │\n",
+
"└─────────────────────────────┴───────────────┴────────────────────────────────────────────────────┘\n"
+ ]
+ }
+ ],
+ "source": [
+ "dt = DeltaTable(table_path)\n",
+ "arrow_table = dt.to_pyarrow_table()\n",
Review Comment:
Probably also worth demonstrating how you can push a query down into the
DeltaLake scan using its `filter` argument (most real world usage would do that
instead of load the entire table into memory and then issue a query on it?).
One of the things you could demo is adding bbox columns on write by using
`ST_Xmin()` and friends. When you read, you can issue a Delta Lake filter like
```
bbox.xmin <= -73.11 AND
bbox.ymin <= 44.03 AND
bbox.xmax >= -73.21 AND
bbox.ymax >= 43.97
```
...except in whatever syntax the delta lake `filter` argument uses. That
should result in a more reasonable fetch from a large local or remote table.
(Otherwise, users are better off just using Parquet because the pushdown is
better).
##########
docs/delta-lake.md:
##########
@@ -0,0 +1,202 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# SedonaDB + Delta Lake
+
+This page shows how to read and write Delta Lake tables with SedonaDB.
+
+Make sure you run `pip install deltalake` to run the cells in this notebook.
+
+
+```python
+from deltalake import write_deltalake, DeltaTable
+import sedona.db
+
+sd = sedona.db.connect()
+```
+
+Read in a GeoParquet dataset into a SedonaDB DataFrame.
+
+
+```python
+countries = sd.read_parquet(
+
"https://raw.githubusercontent.com/geoarrow/geoarrow-data/v0.2.0/natural-earth/files/natural-earth_countries_geo.parquet";
+)
+```
+
+## Create a Delta Lake table
+
+Now write the DataFrame to a Delta Lake table. Notice that you must convert
the geometry column to Well-Known Text (WKT) before writing to the Delta table.
+
+Delta Lake does not support geometry columns.
+
+
+```python
+countries.to_view("countries", True)
+df = sd.sql(
+ "select name, continent, ST_AsText(geometry) as geometry_wkt from
countries"
Review Comment:
In real life you'd probably want `ST_AsBinary()` (more compact for most
usage)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
