dramaticlly commented on code in PR #7499:
URL: https://github.com/apache/iceberg/pull/7499#discussion_r1187714728
##########
docs/spark-writes.md:
##########
@@ -339,74 +331,60 @@ USING iceberg
PARTITIONED BY (days(ts), category)
```
-To write data to the sample table, your data needs to be sorted by `days(ts),
category`.
-
-If you're inserting data with SQL statement, you can use `ORDER BY` to achieve
it, like below:
+To write data to the sample table, your data needs to be sorted by `days(ts),
category` but this is taken care
+of automatically by the default `hash` distribution. Previously this would
have required manually sorting, but this
+is no longer the case.
```sql
INSERT INTO prod.db.sample
SELECT id, data, category, ts FROM another_table
-ORDER BY ts, category
-```
-
-If you're inserting data with DataFrame, you can use either `orderBy`/`sort`
to trigger global sort, or `sortWithinPartitions`
-to trigger local sort. Local sort for example:
-
-```scala
-data.sortWithinPartitions("ts", "category")
- .writeTo("prod.db.sample")
- .append()
```
-You can simply add the original column to the sort condition for the most
partition transformations, except `bucket`.
-
-For `bucket` partition transformation, you need to register the Iceberg
transform function in Spark to specify it during sort.
-
-Let's go through another sample table having bucket partition:
-
-```sql
-CREATE TABLE prod.db.sample (
- id bigint,
- data string,
- category string,
- ts timestamp)
-USING iceberg
-PARTITIONED BY (bucket(16, id))
-```
-
-You need to register the function to deal with bucket, like below:
-
-```scala
-import org.apache.iceberg.spark.IcebergSpark
-import org.apache.spark.sql.types.DataTypes
-
-IcebergSpark.registerBucketUDF(spark, "iceberg_bucket16", DataTypes.LongType,
16)
-```
-
-{{< hint info >}}
-Explicit registration of the function is necessary because Spark doesn't allow
Iceberg to provide functions.
-[SPARK-27658](https://issues.apache.org/jira/browse/SPARK-27658) is filed to
enable Iceberg to provide functions
-which can be used in query.
-{{< /hint >}}
-
-Here we just registered the bucket function as `iceberg_bucket16`, which can
be used in sort clause.
-
-If you're inserting data with SQL statement, you can use the function like
below:
-
-```sql
-INSERT INTO prod.db.sample
-SELECT id, data, category, ts FROM another_table
-ORDER BY iceberg_bucket16(id)
-```
-
-If you're inserting data with DataFrame, you can use the function like below:
-
-```scala
-data.sortWithinPartitions(expr("iceberg_bucket16(id)"))
- .writeTo("prod.db.sample")
- .append()
-```
+There are 3 options for `write.distribution-mode`
+
+* `none` - This is the previous default for Iceberg.
+<p>This mode does not request any shuffles or sort to be performed
automatically by Spark. Because no work is done
+automatically by Spark, the data must be *manually* locally or globally sorted
by partition value. To reduce the number
Review Comment:
if you intend to italicize the word manually here using markdown syntax
`*manually*`, I think it might not work as intended within `<p>`. I think below
will work. Please ignore me if you want literal asterisk
```markdown
<p>This mode does not request any shuffles or sort to be performed
automatically by Spark. Because no work is done
automatically by Spark, the data must be <em>manually</em> locally or
globally sorted by partition value. To reduce the number
of files produced during writing, using a global sort is recommended.</p>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
