ueshin commented on code in PR #52034:
URL: https://github.com/apache/spark/pull/52034#discussion_r2286093973
##########
python/docs/source/user_guide/ansi_migration_guide.ipynb:
##########
@@ -0,0 +1,193 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4fa81d13",
+ "metadata": {},
+ "source": [
+ "# ANSI Migration Guide - Pandas API on Spark\n",
+ "ANSI mode is now on by default for Pandas API on Spark. This guide helps
you understand the key behavior differences you’ll see.\n",
+ "In short, with ANSI mode on, Pandas API on Spark behavior matches native
pandas in cases where Pandas API on Spark with ANSI off did not."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6e1c7952",
+ "metadata": {},
+ "source": [
+ "## Behavior Change\n",
+ "### String Number Comparison\n",
+ "**ANSI off:** Spark implicitly casts numbers and strings, so `1` and
`'1'` are considered equal.\n",
+ "\n",
+ "**ANSI on:** behaves like pandas, `1 == '1'` is False."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "69474e28-c1cd-40fe-8ec6-7373b56c4dee",
+ "metadata": {},
+ "source": [
+ "Examples are as shown below:\n",
+ "\n",
+ "```python\n",
+ ">>> pdf = pd.DataFrame({\"int\": [1, 2], \"str\": [\"1\", \"2\"]})\n",
+ ">>> psdf = ps.from_pandas(pdf)\n",
+ "\n",
+ "# ANSI on\n",
+ ">>> psdf[\"int\"] == psdf[\"str\"]\n",
Review Comment:
Shall we add the configuration here to make it clear?
```suggestion
"# ANSI on\n",
">>> spark.conf.set(\"spark.sql.ansi.enabled\", True)\n",
">>> psdf[\"int\"] == psdf[\"str\"]\n",
```
same for the similar places below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
