This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 913ca2d  Publishing website 2019/12/03 09:33:01 at commit 72a2d04
913ca2d is described below

commit 913ca2d4d8ed65b124c75add3f908de260445f53
Author: jenkins <bui...@apache.org>
AuthorDate: Tue Dec 3 09:33:01 2019 +0000

    Publishing website 2019/12/03 09:33:01 at commit 72a2d04
---
 .../runners/capability-matrix/index.html           | 1258 ++++++++++++++++++--
 .../documentation/runners/spark/index.html         |  168 ++-
 2 files changed, 1305 insertions(+), 121 deletions(-)

diff --git 
a/website/generated-content/documentation/runners/capability-matrix/index.html 
b/website/generated-content/documentation/runners/capability-matrix/index.html
index 83716f5..16a9ab0 100644
--- 
a/website/generated-content/documentation/runners/capability-matrix/index.html
+++ 
b/website/generated-content/documentation/runners/capability-matrix/index.html
@@ -300,7 +300,9 @@ limitations under the License.
   
     <th class="cap-summary color-platform format-platform" 
style="color:#ec3">Apache Flink</th>
   
-    <th class="cap-summary color-platform format-platform" 
style="color:#ec3">Apache Spark</th>
+    <th class="cap-summary color-platform format-platform" 
style="color:#ec3">Apache Spark (RDD/DStream based)</th>
+  
+    <th class="cap-summary color-platform format-platform" 
style="color:#ec3">Apache Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap-summary color-platform format-platform" 
style="color:#ec3">Apache Apex</th>
   
@@ -397,6 +399,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -618,6 +638,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -839,6 +877,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -1060,6 +1116,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -1317,6 +1391,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -1502,6 +1594,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -1723,6 +1833,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -1944,6 +2072,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -2165,6 +2311,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -2368,6 +2532,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -2553,7 +2735,9 @@ limitations under the License.
   
     <th class="cap-summary color-platform format-platform" 
style="color:#59f">Apache Flink</th>
   
-    <th class="cap-summary color-platform format-platform" 
style="color:#59f">Apache Spark</th>
+    <th class="cap-summary color-platform format-platform" 
style="color:#59f">Apache Spark (RDD/DStream based)</th>
+  
+    <th class="cap-summary color-platform format-platform" 
style="color:#59f">Apache Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap-summary color-platform format-platform" 
style="color:#59f">Apache Apex</th>
   
@@ -2650,7 +2834,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -2663,7 +2847,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -2792,11 +2976,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Fixed windows</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -2815,6 +2994,11 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Fixed windows</th>
+    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -2889,7 +3073,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -2902,7 +3086,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -3013,11 +3197,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Sliding windows</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3054,6 +3233,11 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Sliding windows</th>
+    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3128,7 +3312,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -3141,7 +3325,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -3234,11 +3418,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Session windows</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3293,6 +3472,11 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Session windows</th>
+    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3367,7 +3551,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -3380,7 +3564,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -3455,11 +3639,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Custom windows</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3532,6 +3711,11 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Custom windows</th>
+    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3606,7 +3790,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -3619,7 +3803,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -3676,11 +3860,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Custom merging windows</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3771,6 +3950,11 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Custom merging windows</th>
+    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -3845,7 +4029,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -3858,7 +4042,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2713;</b></center>
+<center><b>~</b></center>
 </td>
     
     
@@ -3897,11 +4081,6 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-  </tr>
-  
-  <tr class="cap-summary">
-    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Timestamp control</th>
-    
     
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
@@ -4010,7 +4189,138 @@ limitations under the License.
 <center><b>&#x2713;</b></center>
 </td>
     
-    
+  </tr>
+  
+  <tr class="cap-summary">
+    <th class="cap-summary color-capability format-capability" 
style="color:#59f">Timestamp control</th>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
 
     <td width="25%" class="cap-summary" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
@@ -4143,7 +4453,9 @@ limitations under the License.
   
     <th class="cap-summary color-platform format-platform" 
style="color:#8c6">Apache Flink</th>
   
-    <th class="cap-summary color-platform format-platform" 
style="color:#8c6">Apache Spark</th>
+    <th class="cap-summary color-platform format-platform" 
style="color:#8c6">Apache Spark (RDD/DStream based)</th>
+  
+    <th class="cap-summary color-platform format-platform" 
style="color:#8c6">Apache Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap-summary color-platform format-platform" 
style="color:#8c6">Apache Apex</th>
   
@@ -4222,7 +4534,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -4235,7 +4547,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2715;</b></center>
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
 </td>
     
     
@@ -4443,7 +4773,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -4456,7 +4786,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2715;</b></center>
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
 </td>
     
     
@@ -4682,6 +5030,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -4885,7 +5251,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -4898,7 +5264,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2715;</b></center>
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
 </td>
     
     
@@ -5266,6 +5650,24 @@ limitations under the License.
 <center><b>&#x2715;</b></center>
 </td>
     
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
   </tr>
   
   <tr class="cap-summary">
@@ -5327,7 +5729,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -5340,7 +5742,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>&#x2715;</b></center>
+<center><b>&#x2713;</b></center>
+</td>
+    
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
 </td>
     
     
@@ -5566,6 +5986,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -5769,6 +6207,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -5954,7 +6410,9 @@ limitations under the License.
   
     <th class="cap-summary color-platform format-platform" 
style="color:#d77">Apache Flink</th>
   
-    <th class="cap-summary color-platform format-platform" 
style="color:#d77">Apache Spark</th>
+    <th class="cap-summary color-platform format-platform" 
style="color:#d77">Apache Spark (RDD/DStream based)</th>
+  
+    <th class="cap-summary color-platform format-platform" 
style="color:#d77">Apache Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap-summary color-platform format-platform" 
style="color:#d77">Apache Apex</th>
   
@@ -6051,6 +6509,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#faa;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#d77;border-color:#b55"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -6272,6 +6748,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#d77;border-color:#b55"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -6635,6 +7129,24 @@ limitations under the License.
 <center><b>&#x2715;</b></center>
 </td>
     
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
   </tr>
   
   <tr class="cap-summary">
@@ -6660,7 +7172,9 @@ limitations under the License.
   
     <th class="cap-summary color-platform format-platform" 
style="color:#bbb">Apache Flink</th>
   
-    <th class="cap-summary color-platform format-platform" 
style="color:#bbb">Apache Spark</th>
+    <th class="cap-summary color-platform format-platform" 
style="color:#bbb">Apache Spark (RDD/DStream based)</th>
+  
+    <th class="cap-summary color-platform format-platform" 
style="color:#bbb">Apache Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap-summary color-platform format-platform" 
style="color:#bbb">Apache Apex</th>
   
@@ -6881,6 +7395,24 @@ limitations under the License.
 <center><b>&#x2715;</b></center>
 </td>
     
+    
+
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
   </tr>
   
   <tr class="cap-summary">
@@ -7050,6 +7582,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap-summary" 
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>&#x2715;</b></center>
+</td>
+    
+    
+
     <td width="25%" class="cap-summary" 
style="background-color:#ccc;border-color:#aaa"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -7154,7 +7704,9 @@ limitations under the License.
   
     <th class="cap color-platform format-platform" style="color:#ec3">Apache 
Flink</th>
   
-    <th class="cap color-platform format-platform" style="color:#ec3">Apache 
Spark</th>
+    <th class="cap color-platform format-platform" style="color:#ec3">Apache 
Spark (RDD/DStream based)</th>
+  
+    <th class="cap color-platform format-platform" style="color:#ec3">Apache 
Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap color-platform format-platform" style="color:#ec3">Apache 
Apex</th>
   
@@ -7251,6 +7803,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />ParDo 
applies per-element transformations as Spark FlatMapFunction.
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -7472,6 +8042,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using 
Spark's <tt>groupByKey</tt>.
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -7693,6 +8281,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Some 
corner cases like flatten on empty collections are not yet supported.
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -7914,6 +8520,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using 
Spark's <tt>Aggregator</tt> and agg function
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -8148,6 +8772,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
+<center><b>Partially: supported via inlining only in batch 
mode</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
 <center><b>Partially: supported via inlining</b></center><br />
 </td>
     
@@ -8356,6 +8998,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using 
Spark's broadcast variables.
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -8577,6 +9237,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: bounded source only</b></center><br />Using Spark's 
DatasourceV2 API in microbatch mode (Continuous streaming mode is tagged 
experimental in spark and does not support aggregation).
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ec3;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -8798,6 +9476,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No: not implemented</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9019,6 +9715,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: All metric types are supported in batch 
mode.</b></center><br />Only attempted values are supported. No committed 
values for metrics.
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9222,6 +9936,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: full support in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#ca1"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9235,7 +9967,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>No: not implemented</b></center><br />Spark supports per-key state 
with <tt>mapWithState()</tt> so support should be straightforward.
+<center><b>No: not implemented</b></center><br />
 </td>
     
     
@@ -9407,7 +10139,9 @@ limitations under the License.
   
     <th class="cap color-platform format-platform" style="color:#59f">Apache 
Flink</th>
   
-    <th class="cap color-platform format-platform" style="color:#59f">Apache 
Spark</th>
+    <th class="cap color-platform format-platform" style="color:#59f">Apache 
Spark (RDD/DStream based)</th>
+  
+    <th class="cap color-platform format-platform" style="color:#59f">Apache 
Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap color-platform format-platform" style="color:#59f">Apache 
Apex</th>
   
@@ -9504,6 +10238,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9725,6 +10477,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9946,6 +10716,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -10167,6 +10955,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -10388,6 +11194,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -10609,6 +11433,42 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Yes: supported</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -10733,6 +11593,11 @@ limitations under the License.
 <center><b>Yes: supported</b></center><br />
 </td>
     
+  </tr>
+  
+  <tr class="cap">
+    <th class="cap color-capability format-capability" 
style="color:#59f">Timestamp control</th>
+    
     
 
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
@@ -10748,14 +11613,9 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>Yes: supported</b></center><br />
+<center><b>Yes: output timestamp for window panes</b></center><br />For a 
grouping transform, such as GBK or Combine, an OutputTimeFn specifies (1) how 
to combine input timestamps within a window and (2) how to merge aggregated 
timestamps when windows merge.
 </td>
     
-  </tr>
-  
-  <tr class="cap">
-    <th class="cap color-capability format-capability" 
style="color:#59f">Timestamp control</th>
-    
     
 
     <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
@@ -10771,7 +11631,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>Yes: output timestamp for window panes</b></center><br />For a 
grouping transform, such as GBK or Combine, an OutputTimeFn specifies (1) how 
to combine input timestamps within a window and (2) how to merge aggregated 
timestamps when windows merge.
+<center><b>Yes: supported</b></center><br />
 </td>
     
     
@@ -10812,7 +11672,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap" 
style="background-color:#59f;border-color:#37d"><!--
+    <td width="25%" class="cap" 
style="background-color:#8cf;border-color:#37d"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -10825,7 +11685,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>Yes: supported</b></center><br />
+<center><b>Partially: fully supported in batch mode</b></center><br />
 </td>
     
     
@@ -10997,7 +11857,9 @@ limitations under the License.
   
     <th class="cap color-platform format-platform" style="color:#8c6">Apache 
Flink</th>
   
-    <th class="cap color-platform format-platform" style="color:#8c6">Apache 
Spark</th>
+    <th class="cap color-platform format-platform" style="color:#8c6">Apache 
Spark (RDD/DStream based)</th>
+  
+    <th class="cap color-platform format-platform" style="color:#8c6">Apache 
Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap color-platform format-platform" style="color:#8c6">Apache 
Apex</th>
   
@@ -11076,7 +11938,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -11089,7 +11951,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
 </td>
     
     
@@ -11297,7 +12177,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -11310,7 +12190,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
 </td>
     
     
@@ -11536,6 +12434,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -11739,7 +12655,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -11752,7 +12668,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
 </td>
     
     
@@ -12027,6 +12961,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
+<center><b>No: pending model support</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
 <center><b>No</b></center><br />
 </td>
     
@@ -12181,7 +13133,7 @@ limitations under the License.
     
     
 
-    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+    <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
@@ -12194,7 +13146,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
 </td>
     
     
@@ -12420,6 +13390,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No: no streaming support in the runner</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#8c6;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -12623,6 +13611,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#6a4"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -12808,7 +13814,9 @@ limitations under the License.
   
     <th class="cap color-platform format-platform" style="color:#d77">Apache 
Flink</th>
   
-    <th class="cap color-platform format-platform" style="color:#d77">Apache 
Spark</th>
+    <th class="cap color-platform format-platform" style="color:#d77">Apache 
Spark (RDD/DStream based)</th>
+  
+    <th class="cap color-platform format-platform" style="color:#d77">Apache 
Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap color-platform format-platform" style="color:#d77">Apache 
Apex</th>
   
@@ -12905,6 +13913,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#faa;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#d77;border-color:#b55"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -13126,6 +14152,24 @@ limitations under the License.
     
     
 
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No</b></center><br />
+</td>
+    
+    
+
     <td width="25%" class="cap" 
style="background-color:#d77;border-color:#b55"><!--
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -13396,6 +14440,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
+<center><b>No: pending model support</b></center><br />
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
 <center><b>No</b></center><br />
 </td>
     
@@ -13514,7 +14576,9 @@ limitations under the License.
   
     <th class="cap color-platform format-platform" style="color:#bbb">Apache 
Flink</th>
   
-    <th class="cap color-platform format-platform" style="color:#bbb">Apache 
Spark</th>
+    <th class="cap color-platform format-platform" style="color:#bbb">Apache 
Spark (RDD/DStream based)</th>
+  
+    <th class="cap color-platform format-platform" style="color:#bbb">Apache 
Spark Structured Streaming (Dataset based)</th>
   
     <th class="cap color-platform format-platform" style="color:#bbb">Apache 
Apex</th>
   
@@ -13735,6 +14799,24 @@ limitations under the License.
 <center><b>: </b></center><br />
 </td>
     
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>: </b></center><br />
+</td>
+    
   </tr>
   
   <tr class="cap">
@@ -13827,6 +14909,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 
express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->
+<center><b>No: </b></center><br />not implemented
+</td>
+    
+    
+
+    <td width="25%" class="cap" 
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
 <center><b>: </b></center><br />
 </td>
     
diff --git a/website/generated-content/documentation/runners/spark/index.html 
b/website/generated-content/documentation/runners/spark/index.html
index 3da651c..f26f634 100644
--- a/website/generated-content/documentation/runners/spark/index.html
+++ b/website/generated-content/documentation/runners/spark/index.html
@@ -201,7 +201,8 @@
 
 
 <ul class="nav">
-  <li><a href="#portability">Portability</a></li>
+  <li><a href="#three-flavors-of-the-spark-runner">Three flavors of the Spark 
runner</a></li>
+  <li><a href="#which-runner-to-use-portable-or-non-portable-runner">Which 
runner to use: portable or non portable runner?</a></li>
   <li><a href="#spark-runner-prerequisites-and-setup">Spark Runner 
prerequisites and setup</a>
     <ul>
       <li><a href="#deploying-spark-with-your-application">Deploying Spark 
with your application</a></li>
@@ -253,44 +254,50 @@ The Spark Runner can execute Spark pipelines just like a 
native Spark applicatio
 
 <p>The <a href="/documentation/runners/capability-matrix/">Beam Capability 
Matrix</a> documents the currently supported capabilities of the Spark 
Runner.</p>
 
-<p><em><strong>Note:</strong></em> <em>support for the Beam Model in streaming 
is currently experimental, follow-up in the <a 
href="/get-started/support/">mailing list</a> for status updates.</em></p>
-
-<h2 id="portability">Portability</h2>
-
-<p>The Spark runner comes in two flavors:</p>
+<h2 id="three-flavors-of-the-spark-runner">Three flavors of the Spark 
runner</h2>
+<p>The Spark runner comes in three flavors:</p>
 
 <ol>
-  <li>A <em>legacy Runner</em> which supports only Java (and other JVM-based 
languages)</li>
+  <li>A <em>legacy Runner</em> which supports only Java (and other JVM-based 
languages) and that is based on Spark RDD/DStream</li>
+  <li>An <em>Structured Streaming Spark Runner</em> which supports only Java 
(and other JVM-based languages) and that is based on Spark Datasets and the <a 
href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html";>Apache
 Spark Structured Streaming</a> framework.
+    <blockquote>
+      <p><strong>Note:</strong> It is still experimental, its coverage of the 
Beam model is partial. As for now it only supports batch mode.</p>
+    </blockquote>
+  </li>
   <li>A <em>portable Runner</em> which supports Java, Python, and Go</li>
 </ol>
 
+<p>This guide is split into two parts to document the non-portable and
+the portable functionality of the Spark Runner. Please use the switcher below 
to
+select the appropriate Runner:</p>
+
+<h2 id="which-runner-to-use-portable-or-non-portable-runner">Which runner to 
use: portable or non portable runner?</h2>
+
 <p>Beam and its Runners originally only supported JVM-based languages
 (e.g. Java/Scala/Kotlin). Python and Go SDKs were added later on. The
 architecture of the Runners had to be changed significantly to support 
executing
 pipelines written in other languages.</p>
 
-<p>If your applications only use Java, then you should currently go with the 
legacy
-Runner. If you want to run Python or Go pipelines with Beam on Spark, you need 
to use
+<p>If your applications only use Java, then you should currently go with one 
of the java based runners.
+If you want to run Python or Go pipelines with Beam on Spark, you need to use
 the portable Runner. For more information on portability, please visit the
 <a href="/roadmap/portability/">Portability page</a>.</p>
 
-<p>This guide is split into two parts to document the legacy and
-the portable functionality of the Spark Runner. Please use the switcher below 
to
-select the appropriate Runner:</p>
-
 <nav class="language-switcher">
   <strong>Adapt for:</strong>
   <ul>
-    <li data-type="language-java">Legacy (Java)</li>
+    <li data-type="language-java">Non portable (Java)</li>
     <li data-type="language-py">Portable (Java/Python/Go)</li>
   </ul>
 </nav>
 
 <h2 id="spark-runner-prerequisites-and-setup">Spark Runner prerequisites and 
setup</h2>
 
-<p>The Spark runner currently supports Spark’s 2.x branch, and more 
specifically any version greater than 2.2.0.</p>
+<p>The Spark runner currently supports Spark’s 2.x branch, and more 
specifically any version greater than 2.4.0.</p>
 
-<p><span class="language-java">You can add a dependency on the latest version 
of the Spark runner by adding to your pom.xml the following:</span></p>
+<p><span class="language-java">
+You can add a dependency on the latest version of the Spark runner by adding 
to your pom.xml the following:
+</span></p>
 
 <div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="o">&lt;</span><span 
class="n">dependency</span><span class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">groupId</span><span 
class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span 
class="na">apache</span><span class="o">.</span><span 
class="na">beam</span><span class="o">&lt;/</span><span 
class="n">groupId</span><span class="o">&gt;</span>
@@ -301,22 +308,26 @@ select the appropriate Runner:</p>
 
 <h3 id="deploying-spark-with-your-application">Deploying Spark with your 
application</h3>
 
-<p><span class="language-java">In some cases, such as running in local 
mode/Standalone, your (self-contained) application would be required to pack 
Spark by explicitly adding the following dependencies in your 
pom.xml:</span></p>
+<p><span class="language-java">
+In some cases, such as running in local mode/Standalone, your (self-contained) 
application would be required to pack Spark by explicitly adding the following 
dependencies in your pom.xml:
+</span></p>
 
 <div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="o">&lt;</span><span 
class="n">dependency</span><span class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">groupId</span><span 
class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span 
class="na">apache</span><span class="o">.</span><span 
class="na">spark</span><span class="o">&lt;/</span><span 
class="n">groupId</span><span class="o">&gt;</span>
-  <span class="o">&lt;</span><span class="n">artifactId</span><span 
class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span 
class="n">core_2</span><span class="o">.</span><span class="mi">10</span><span 
class="o">&lt;/</span><span class="n">artifactId</span><span 
class="o">&gt;</span>
+  <span class="o">&lt;</span><span class="n">artifactId</span><span 
class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span 
class="n">core_2</span><span class="o">.</span><span class="mi">11</span><span 
class="o">&lt;/</span><span class="n">artifactId</span><span 
class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">version</span><span 
class="o">&gt;</span><span class="err">$</span><span class="o">{</span><span 
class="n">spark</span><span class="o">.</span><span 
class="na">version</span><span class="o">}&lt;/</span><span 
class="n">version</span><span class="o">&gt;</span>
 <span class="o">&lt;/</span><span class="n">dependency</span><span 
class="o">&gt;</span>
 
 <span class="o">&lt;</span><span class="n">dependency</span><span 
class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">groupId</span><span 
class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span 
class="na">apache</span><span class="o">.</span><span 
class="na">spark</span><span class="o">&lt;/</span><span 
class="n">groupId</span><span class="o">&gt;</span>
-  <span class="o">&lt;</span><span class="n">artifactId</span><span 
class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span 
class="n">streaming_2</span><span class="o">.</span><span 
class="mi">10</span><span class="o">&lt;/</span><span 
class="n">artifactId</span><span class="o">&gt;</span>
+  <span class="o">&lt;</span><span class="n">artifactId</span><span 
class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span 
class="n">streaming_2</span><span class="o">.</span><span 
class="mi">11</span><span class="o">&lt;/</span><span 
class="n">artifactId</span><span class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">version</span><span 
class="o">&gt;</span><span class="err">$</span><span class="o">{</span><span 
class="n">spark</span><span class="o">.</span><span 
class="na">version</span><span class="o">}&lt;/</span><span 
class="n">version</span><span class="o">&gt;</span>
 <span class="o">&lt;/</span><span class="n">dependency</span><span 
class="o">&gt;</span>
 </code></pre></div></div>
 
-<p><span class="language-java">And shading the application jar using the maven 
shade plugin:</span></p>
+<p><span class="language-java">
+And shading the application jar using the maven shade plugin:
+</span></p>
 
 <div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="o">&lt;</span><span 
class="n">plugin</span><span class="o">&gt;</span>
   <span class="o">&lt;</span><span class="n">groupId</span><span 
class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span 
class="na">apache</span><span class="o">.</span><span 
class="na">maven</span><span class="o">.</span><span 
class="na">plugins</span><span class="o">&lt;/</span><span 
class="n">groupId</span><span class="o">&gt;</span>
@@ -353,17 +364,30 @@ select the appropriate Runner:</p>
 <span class="o">&lt;/</span><span class="n">plugin</span><span 
class="o">&gt;</span>
 </code></pre></div></div>
 
-<p><span class="language-java">After running <code>mvn package</code>, run 
<code>ls target</code> and you should see (assuming your artifactId is <code 
class="highlighter-rouge">beam-examples</code> and the version is <code 
class="highlighter-rouge">1.0.0</code>):</span></p>
+<p><span class="language-java">
+After running <code>mvn package</code>, run <code>ls target</code> and you 
should see (assuming your artifactId is <code 
class="highlighter-rouge">beam-examples</code> and the version is <code 
class="highlighter-rouge">1.0.0</code>):
+</span></p>
+
+<div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="n">beam</span><span 
class="o">-</span><span class="n">examples</span><span class="o">-</span><span 
class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span 
class="o">-</span><span class="n">shaded</span><span class="o">.</span><span 
class="na">jar</span>
+</code></pre></div></div>
+
+<p><span class="language-java">
+To run against a Standalone cluster simply run:
+</span></p>
 
-<p><code class="language-java">
-beam-examples-1.0.0-shaded.jar
-</code></p>
+<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
+</span></p>
 
-<p><span class="language-java">To run against a Standalone cluster simply 
run:</span></p>
+<div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="n">spark</span><span 
class="o">-</span><span class="n">submit</span> <span class="o">--</span><span 
class="kd">class</span> <span class="nc">com</span><span 
class="o">.</span><span class="na">beam</span><span class="o">.</span><span 
class="na">examples</span><span class="o">.</span><span 
class="na">BeamPipeline</span> <span class="o">--</span><span 
class="n">master</span> <span c [...]
+</code></pre></div></div>
 
-<p><code class="language-java">
-spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT 
target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
-</code></p>
+<p><span class="language-java">
+<br /><b>For Structured Streaming based runner:</b><br />
+</span></p>
+
+<div class="language-java highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="n">spark</span><span 
class="o">-</span><span class="n">submit</span> <span class="o">--</span><span 
class="kd">class</span> <span class="nc">com</span><span 
class="o">.</span><span class="na">beam</span><span class="o">.</span><span 
class="na">examples</span><span class="o">.</span><span 
class="na">BeamPipeline</span> <span class="o">--</span><span 
class="n">master</span> <span c [...]
+</code></pre></div></div>
 
 <p><span class="language-py">
 You will need Docker to be installed in your execution environment. To develop
@@ -381,8 +405,7 @@ download it on the <a 
href="/get-started/downloads/">Downloads page</a>. In the
 available.
 </span></p>
 
-<p><span class="language-py">1. Start the JobService endpoint: <code 
class="highlighter-rouge">./gradlew :runners:spark:job-server:runShadow</code>
-</span></p>
+<p><span class="language-py">1. Start the JobService endpoint: <code 
class="highlighter-rouge">./gradlew 
:runners:spark:job-server:runShadow</code></span></p>
 
 <p><span class="language-py">
 The JobService is the central instance where you submit your Beam pipeline.
@@ -391,8 +414,7 @@ job. To execute the job on a Spark cluster, the Beam 
JobService needs to be
 provided with the Spark master address.
 </span></p>
 
-<p><span class="language-py">2. Submit the Python pipeline to the above 
endpoint by using the <code class="highlighter-rouge">PortableRunner</code>, 
<code class="highlighter-rouge">job_endpoint</code> set to <code 
class="highlighter-rouge">localhost:8099</code> (this is the default address of 
the JobService), and <code class="highlighter-rouge">environment_type</code> 
set to <code class="highlighter-rouge">LOOPBACK</code>. For example:
-</span></p>
+<p><span class="language-py">2. Submit the Python pipeline to the above 
endpoint by using the <code class="highlighter-rouge">PortableRunner</code>, 
<code class="highlighter-rouge">job_endpoint</code> set to <code 
class="highlighter-rouge">localhost:8099</code> (this is the default address of 
the JobService), and <code class="highlighter-rouge">environment_type</code> 
set to <code class="highlighter-rouge">LOOPBACK</code>. For example:</span></p>
 
 <div class="language-py highlighter-rouge"><div class="highlight"><pre 
class="highlight"><code><span class="kn">import</span> <span 
class="nn">apache_beam</span> <span class="k">as</span> <span 
class="n">beam</span>
 <span class="kn">from</span> <span 
class="nn">apache_beam.options.pipeline_options</span> <span 
class="kn">import</span> <span class="n">PipelineOptions</span>
@@ -411,16 +433,13 @@ provided with the Spark master address.
 <p>Deploying your Beam pipeline on a cluster that already has a Spark 
deployment (Spark classes are available in container classpath) does not 
require any additional dependencies.
 For more details on the different deployment modes see: <a 
href="http://spark.apache.org/docs/latest/spark-standalone.html";>Standalone</a>,
 <a href="http://spark.apache.org/docs/latest/running-on-yarn.html";>YARN</a>, 
or <a 
href="http://spark.apache.org/docs/latest/running-on-mesos.html";>Mesos</a>.</p>
 
-<p><span class="language-py">1. Start a Spark cluster which exposes the master 
on port 7077 by default.
-</span></p>
+<p><span class="language-py">1. Start a Spark cluster which exposes the master 
on port 7077 by default.</span></p>
 
-<p><span class="language-py">2. Start JobService that will connect with the 
Spark master: <code class="highlighter-rouge">./gradlew 
:runners:spark:job-server:runShadow 
-PsparkMasterUrl=spark://localhost:7077</code>.
-</span></p>
+<p><span class="language-py">2. Start JobService that will connect with the 
Spark master: <code class="highlighter-rouge">./gradlew 
:runners:spark:job-server:runShadow 
-PsparkMasterUrl=spark://localhost:7077</code>.</span></p>
 
 <p><span class="language-py">3. Submit the pipeline as above.
 Note however that <code 
class="highlighter-rouge">environment_type=LOOPBACK</code> is only intended for 
local testing.
-See <a href="/roadmap/portability/#sdk-harness-config">here</a> for details.
-</span></p>
+See <a href="/roadmap/portability/#sdk-harness-config">here</a> for 
details.</span></p>
 
 <p><span class="language-py">
 (Note that, depending on your cluster setup, you may need to change the <code 
class="highlighter-rouge">environment_type</code> option.
@@ -431,6 +450,10 @@ See <a 
href="/roadmap/portability/#sdk-harness-config">here</a> for details.)
 
 <p>When executing your pipeline with the Spark Runner, you should consider the 
following pipeline options.</p>
 
+<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
+</span></p>
+
 <table class="language-java table table-bordered">
 <tr>
   <th>Field</th>
@@ -469,6 +492,53 @@ See <a 
href="/roadmap/portability/#sdk-harness-config">here</a> for details.)
 </tr>
 </table>
 
+<p><span class="language-java">
+<br /><b>For Structured Streaming based runner:</b><br />
+</span></p>
+
+<table class="language-java table table-bordered">
+<tr>
+  <th>Field</th>
+  <th>Description</th>
+  <th>Default Value</th>
+</tr>
+<tr>
+  <td><code>runner</code></td>
+  <td>The pipeline runner to use. This option allows you to determine the 
pipeline runner at runtime.</td>
+  <td>Set to <code>SparkStructuredStreamingRunner</code> to run using Spark 
Structured Streaming.</td>
+</tr>
+<tr>
+  <td><code>sparkMaster</code></td>
+  <td>The url of the Spark Master. This is the equivalent of setting 
<code>SparkConf#setMaster(String)</code> and can either be 
<code>local[x]</code> to run local with x cores, <code>spark://host:port</code> 
to connect to a Spark Standalone cluster, <code>mesos://host:port</code> to 
connect to a Mesos cluster, or <code>yarn</code> to connect to a yarn 
cluster.</td>
+  <td><code>local[4]</code></td>
+</tr>
+<tr>
+  <td><code>testMode</code></td>
+  <td>Enable test mode that gives useful debugging information: catalyst 
execution plans and Beam DAG printing</td>
+  <td>false</td>
+</tr>
+<tr>
+  <td><code>enableSparkMetricSinks</code></td>
+  <td>Enable reporting metrics to Spark's metrics Sinks.</td>
+  <td>true</td>
+</tr>
+<tr>
+  <td><code>checkpointDir</code></td>
+  <td>A checkpoint directory for streaming resilience, ignored in batch. For 
durability, a reliable filesystem such as HDFS/S3/GS is necessary.</td>
+  <td>local dir in /tmp</td>
+</tr>
+<tr>
+  <td><code>filesToStage</code></td>
+  <td>Jar-Files to send to all workers and put on the classpath.</td>
+  <td>all files from the classpath</td>
+</tr>
+<tr>
+  <td><code>EnableSparkMetricSinks</code></td>
+  <td>Enable/disable sending aggregator values to Spark's metric sinks</td>
+  <td>true</td>
+</tr>
+</table>
+
 <table class="language-py table table-bordered">
 <tr>
   <th>Field</th>
@@ -509,17 +579,31 @@ Spark provides a <a 
href="http://spark.apache.org/docs/latest/monitoring.html#me
 <h3 id="streaming-execution">Streaming Execution</h3>
 
 <p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
 If your pipeline uses an <code>UnboundedSource</code> the Spark Runner will 
automatically set streaming mode. Forcing streaming mode is mostly used for 
testing and is not recommended.
-</span>
-<span class="language-py">Streaming is not yet supported on the Spark portable 
runner.</span></p>
+<br />
+<br /><b>For Structured Streaming based runner:</b><br />
+Streaming mode is not implemented yet in the Spark Structured Streaming runner.
+</span></p>
+
+<p><span class="language-py">
+Streaming is not yet supported on the Spark portable runner.
+</span></p>
 
 <h3 id="using-a-provided-sparkcontext-and-streaminglisteners">Using a provided 
SparkContext and StreamingListeners</h3>
 
 <p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
 If you would like to execute your Spark job with a provided 
<code>SparkContext</code>, such as when using the <a 
href="https://github.com/spark-jobserver/spark-jobserver";>spark-jobserver</a>, 
or use <code>StreamingListeners</code>, you can’t use 
<code>SparkPipelineOptions</code> (the context or a listener cannot be passed 
as a command-line argument anyway).
 Instead, you should use <code>SparkContextOptions</code> which can only be 
used programmatically and is not a common <code>PipelineOptions</code> 
implementation.
-</span>
-<span class="language-py">Provided SparkContext and StreamingListeners are not 
supported on the Spark portable runner.</span></p>
+<br />
+<br /><b>For Structured Streaming based runner:</b><br />
+Provided SparkSession and StreamingListeners are not supported on the Spark 
Structured Streaming runner
+</span></p>
+
+<p><span class="language-py">
+Provided SparkContext and StreamingListeners are not supported on the Spark 
portable runner.
+</span></p>
 
       </div>
     </div>

Reply via email to