Re: [PR] [spark] Introduce spark compact procedure [incubator-paimon]

Mon, 13 Nov 2023 23:55:48 -0800 


Zouxxyy commented on code in PR #2296:
URL: https://github.com/apache/incubator-paimon/pull/2296#discussion_r1392125881


##########
docs/content/engines/spark3.md:
##########
@@ -425,6 +425,29 @@ val query = spark.readStream
 */
 ```
 
+## Spark Procedure
+
+This section introduce all available spark procedures about paimon.
+
+<table class="table table-bordered">
+    <thead>
+    <tr>
+      <th class="text-left" style="width: 4%">Procedure Name</th>
+      <th class="text-left" style="width: 4%">Usage</th>
+      <th class="text-left" style="width: 20%">Explaination</th>
+      <th class="text-left" style="width: 4%">Example</th>
+    </tr>
+    </thead>
+    <tbody style="font-size: 12px; ">
+    <tr>
+      <td>compact</td>
+      <td><nobr>CALL 
compact('&ltidentifier&gt','&ltpartitions&gt','&ltsort_type&gt','&ltcolumns&gt')</nobr><br>CALL
 compact(table => '&ltidentifier&gt' [,partitions => '&ltpartitions&gt'] [, 
order_strategy =>'&ltsort_type&gt'] [,order_by => '&ltcolumns&gt'])</td>
+      <td>identifier: the target table identifier<br><br><nobr>partitions: 
partition filter<br> "," means "AND"<br>";" means 
"OR"</nobr><br><br>order_strategy: 'order' or 'zorder' or 'none' 
<br><br><nobr>order_columns: the columns need to be sort</nobr><br><br>If you 
want sort compact two partitions date=01 and date=02, you need to write 
'date=01;date=02'<br><br>If you want sort one partition with date=01 and 
day=01, you need to write 'date=01,day=01'</td>
+      <td><nobr>SET spark.sql.shuffle.partitions=10; --set the sort 
parallelism</nobr> <nobr>CALL 
paimon.sys.compact('my_db.Orders1','f0=0,f1=1;f0=1,f1=1', 'zorder', 
'f1,f2');</nobr><br><nobr>CALL paimon.sys.compact(table => 'T', partitions => 
'p=0',  order_strategy => 'zorder', order_by => 'a,b')</nobr></td>

Review Comment:
   I don't think omitting parameter name is a good way to use procedures, it's 
not clear enough and does not support order swapping. Although flink currently 
does not support (parameterName => value) ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to