This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 913ca2d Publishing website 2019/12/03 09:33:01 at commit 72a2d04
913ca2d is described below
commit 913ca2d4d8ed65b124c75add3f908de260445f53
Author: jenkins <[email protected]>
AuthorDate: Tue Dec 3 09:33:01 2019 +0000
Publishing website 2019/12/03 09:33:01 at commit 72a2d04
---
.../runners/capability-matrix/index.html | 1258 ++++++++++++++++++--
.../documentation/runners/spark/index.html | 168 ++-
2 files changed, 1305 insertions(+), 121 deletions(-)
diff --git
a/website/generated-content/documentation/runners/capability-matrix/index.html
b/website/generated-content/documentation/runners/capability-matrix/index.html
index 83716f5..16a9ab0 100644
---
a/website/generated-content/documentation/runners/capability-matrix/index.html
+++
b/website/generated-content/documentation/runners/capability-matrix/index.html
@@ -300,7 +300,9 @@ limitations under the License.
<th class="cap-summary color-platform format-platform"
style="color:#ec3">Apache Flink</th>
- <th class="cap-summary color-platform format-platform"
style="color:#ec3">Apache Spark</th>
+ <th class="cap-summary color-platform format-platform"
style="color:#ec3">Apache Spark (RDD/DStream based)</th>
+
+ <th class="cap-summary color-platform format-platform"
style="color:#ec3">Apache Spark Structured Streaming (Dataset based)</th>
<th class="cap-summary color-platform format-platform"
style="color:#ec3">Apache Apex</th>
@@ -397,6 +399,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -618,6 +638,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -839,6 +877,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -1060,6 +1116,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -1317,6 +1391,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -1502,6 +1594,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -1723,6 +1833,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -1944,6 +2072,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -2165,6 +2311,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -2368,6 +2532,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -2553,7 +2735,9 @@ limitations under the License.
<th class="cap-summary color-platform format-platform"
style="color:#59f">Apache Flink</th>
- <th class="cap-summary color-platform format-platform"
style="color:#59f">Apache Spark</th>
+ <th class="cap-summary color-platform format-platform"
style="color:#59f">Apache Spark (RDD/DStream based)</th>
+
+ <th class="cap-summary color-platform format-platform"
style="color:#59f">Apache Spark Structured Streaming (Dataset based)</th>
<th class="cap-summary color-platform format-platform"
style="color:#59f">Apache Apex</th>
@@ -2650,7 +2834,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -2663,7 +2847,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -2792,11 +2976,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Fixed windows</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -2815,6 +2994,11 @@ limitations under the License.
<center><b>✓</b></center>
</td>
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Fixed windows</th>
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -2889,7 +3073,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -2902,7 +3086,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -3013,11 +3197,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Sliding windows</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3054,6 +3233,11 @@ limitations under the License.
<center><b>✓</b></center>
</td>
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Sliding windows</th>
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3128,7 +3312,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -3141,7 +3325,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -3234,11 +3418,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Session windows</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3293,6 +3472,11 @@ limitations under the License.
<center><b>✓</b></center>
</td>
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Session windows</th>
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3367,7 +3551,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -3380,7 +3564,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -3455,11 +3639,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Custom windows</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3532,6 +3711,11 @@ limitations under the License.
<center><b>✓</b></center>
</td>
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Custom windows</th>
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3606,7 +3790,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -3619,7 +3803,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -3676,11 +3860,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Custom merging windows</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3771,6 +3950,11 @@ limitations under the License.
<center><b>✓</b></center>
</td>
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Custom merging windows</th>
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -3845,7 +4029,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -3858,7 +4042,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✓</b></center>
+<center><b>~</b></center>
</td>
@@ -3897,11 +4081,6 @@ limitations under the License.
<center><b>✓</b></center>
</td>
- </tr>
-
- <tr class="cap-summary">
- <th class="cap-summary color-capability format-capability"
style="color:#59f">Timestamp control</th>
-
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
@@ -4010,7 +4189,138 @@ limitations under the License.
<center><b>✓</b></center>
</td>
-
+ </tr>
+
+ <tr class="cap-summary">
+ <th class="cap-summary color-capability format-capability"
style="color:#59f">Timestamp control</th>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✓</b></center>
+</td>
+
+
<td width="25%" class="cap-summary"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -4143,7 +4453,9 @@ limitations under the License.
<th class="cap-summary color-platform format-platform"
style="color:#8c6">Apache Flink</th>
- <th class="cap-summary color-platform format-platform"
style="color:#8c6">Apache Spark</th>
+ <th class="cap-summary color-platform format-platform"
style="color:#8c6">Apache Spark (RDD/DStream based)</th>
+
+ <th class="cap-summary color-platform format-platform"
style="color:#8c6">Apache Spark Structured Streaming (Dataset based)</th>
<th class="cap-summary color-platform format-platform"
style="color:#8c6">Apache Apex</th>
@@ -4222,7 +4534,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -4235,7 +4547,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✕</b></center>
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
</td>
@@ -4443,7 +4773,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -4456,7 +4786,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✕</b></center>
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
</td>
@@ -4682,6 +5030,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -4885,7 +5251,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -4898,7 +5264,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✕</b></center>
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
</td>
@@ -5266,6 +5650,24 @@ limitations under the License.
<center><b>✕</b></center>
</td>
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
</tr>
<tr class="cap-summary">
@@ -5327,7 +5729,7 @@ limitations under the License.
- <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -5340,7 +5742,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>✕</b></center>
+<center><b>✓</b></center>
+</td>
+
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
</td>
@@ -5566,6 +5986,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -5769,6 +6207,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -5954,7 +6410,9 @@ limitations under the License.
<th class="cap-summary color-platform format-platform"
style="color:#d77">Apache Flink</th>
- <th class="cap-summary color-platform format-platform"
style="color:#d77">Apache Spark</th>
+ <th class="cap-summary color-platform format-platform"
style="color:#d77">Apache Spark (RDD/DStream based)</th>
+
+ <th class="cap-summary color-platform format-platform"
style="color:#d77">Apache Spark Structured Streaming (Dataset based)</th>
<th class="cap-summary color-platform format-platform"
style="color:#d77">Apache Apex</th>
@@ -6051,6 +6509,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#faa;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>~</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#d77;border-color:#b55"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -6272,6 +6748,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#d77;border-color:#b55"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -6635,6 +7129,24 @@ limitations under the License.
<center><b>✕</b></center>
</td>
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
</tr>
<tr class="cap-summary">
@@ -6660,7 +7172,9 @@ limitations under the License.
<th class="cap-summary color-platform format-platform"
style="color:#bbb">Apache Flink</th>
- <th class="cap-summary color-platform format-platform"
style="color:#bbb">Apache Spark</th>
+ <th class="cap-summary color-platform format-platform"
style="color:#bbb">Apache Spark (RDD/DStream based)</th>
+
+ <th class="cap-summary color-platform format-platform"
style="color:#bbb">Apache Spark Structured Streaming (Dataset based)</th>
<th class="cap-summary color-platform format-platform"
style="color:#bbb">Apache Apex</th>
@@ -6881,6 +7395,24 @@ limitations under the License.
<center><b>✕</b></center>
</td>
+
+
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
</tr>
<tr class="cap-summary">
@@ -7050,6 +7582,24 @@ limitations under the License.
+ <td width="25%" class="cap-summary"
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>✕</b></center>
+</td>
+
+
+
<td width="25%" class="cap-summary"
style="background-color:#ccc;border-color:#aaa"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -7154,7 +7704,9 @@ limitations under the License.
<th class="cap color-platform format-platform" style="color:#ec3">Apache
Flink</th>
- <th class="cap color-platform format-platform" style="color:#ec3">Apache
Spark</th>
+ <th class="cap color-platform format-platform" style="color:#ec3">Apache
Spark (RDD/DStream based)</th>
+
+ <th class="cap color-platform format-platform" style="color:#ec3">Apache
Spark Structured Streaming (Dataset based)</th>
<th class="cap color-platform format-platform" style="color:#ec3">Apache
Apex</th>
@@ -7251,6 +7803,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />ParDo
applies per-element transformations as Spark FlatMapFunction.
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -7472,6 +8042,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using
Spark's <tt>groupByKey</tt>.
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -7693,6 +8281,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Some
corner cases like flatten on empty collections are not yet supported.
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -7914,6 +8520,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using
Spark's <tt>Aggregator</tt> and agg function
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -8148,6 +8772,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
+<center><b>Partially: supported via inlining only in batch
mode</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
<center><b>Partially: supported via inlining</b></center><br />
</td>
@@ -8356,6 +8998,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />Using
Spark's broadcast variables.
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -8577,6 +9237,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: bounded source only</b></center><br />Using Spark's
DatasourceV2 API in microbatch mode (Continuous streaming mode is tagged
experimental in spark and does not support aggregation).
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ec3;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -8798,6 +9476,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No: not implemented</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -9019,6 +9715,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: All metric types are supported in batch
mode.</b></center><br />Only attempted values are supported. No committed
values for metrics.
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ddd;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -9222,6 +9936,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#fe5;border-color:#ca1"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: full support in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ddd;border-color:#ca1"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -9235,7 +9967,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>No: not implemented</b></center><br />Spark supports per-key state
with <tt>mapWithState()</tt> so support should be straightforward.
+<center><b>No: not implemented</b></center><br />
</td>
@@ -9407,7 +10139,9 @@ limitations under the License.
<th class="cap color-platform format-platform" style="color:#59f">Apache
Flink</th>
- <th class="cap color-platform format-platform" style="color:#59f">Apache
Spark</th>
+ <th class="cap color-platform format-platform" style="color:#59f">Apache
Spark (RDD/DStream based)</th>
+
+ <th class="cap color-platform format-platform" style="color:#59f">Apache
Spark Structured Streaming (Dataset based)</th>
<th class="cap color-platform format-platform" style="color:#59f">Apache
Apex</th>
@@ -9504,6 +10238,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -9725,6 +10477,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -9946,6 +10716,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -10167,6 +10955,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -10388,6 +11194,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -10609,6 +11433,42 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Yes: supported</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -10733,6 +11593,11 @@ limitations under the License.
<center><b>Yes: supported</b></center><br />
</td>
+ </tr>
+
+ <tr class="cap">
+ <th class="cap color-capability format-capability"
style="color:#59f">Timestamp control</th>
+
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
@@ -10748,14 +11613,9 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>Yes: supported</b></center><br />
+<center><b>Yes: output timestamp for window panes</b></center><br />For a
grouping transform, such as GBK or Combine, an OutputTimeFn specifies (1) how
to combine input timestamps within a window and (2) how to merge aggregated
timestamps when windows merge.
</td>
- </tr>
-
- <tr class="cap">
- <th class="cap color-capability format-capability"
style="color:#59f">Timestamp control</th>
-
<td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
@@ -10771,7 +11631,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>Yes: output timestamp for window panes</b></center><br />For a
grouping transform, such as GBK or Combine, an OutputTimeFn specifies (1) how
to combine input timestamps within a window and (2) how to merge aggregated
timestamps when windows merge.
+<center><b>Yes: supported</b></center><br />
</td>
@@ -10812,7 +11672,7 @@ limitations under the License.
- <td width="25%" class="cap"
style="background-color:#59f;border-color:#37d"><!--
+ <td width="25%" class="cap"
style="background-color:#8cf;border-color:#37d"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -10825,7 +11685,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>Yes: supported</b></center><br />
+<center><b>Partially: fully supported in batch mode</b></center><br />
</td>
@@ -10997,7 +11857,9 @@ limitations under the License.
<th class="cap color-platform format-platform" style="color:#8c6">Apache
Flink</th>
- <th class="cap color-platform format-platform" style="color:#8c6">Apache
Spark</th>
+ <th class="cap color-platform format-platform" style="color:#8c6">Apache
Spark (RDD/DStream based)</th>
+
+ <th class="cap color-platform format-platform" style="color:#8c6">Apache
Spark Structured Streaming (Dataset based)</th>
<th class="cap color-platform format-platform" style="color:#8c6">Apache
Apex</th>
@@ -11076,7 +11938,7 @@ limitations under the License.
- <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -11089,7 +11951,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
</td>
@@ -11297,7 +12177,7 @@ limitations under the License.
- <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -11310,7 +12190,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
</td>
@@ -11536,6 +12434,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -11739,7 +12655,7 @@ limitations under the License.
- <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -11752,7 +12668,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
</td>
@@ -12027,6 +12961,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
+<center><b>No: pending model support</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
<center><b>No</b></center><br />
</td>
@@ -12181,7 +13133,7 @@ limitations under the License.
- <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+ <td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -12194,7 +13146,25 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<center><b>No</b></center><br />
+<center><b>Yes: fully supported</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
</td>
@@ -12420,6 +13390,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No: no streaming support in the runner</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#8c6;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -12623,6 +13611,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#ae8;border-color:#6a4"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#ddd;border-color:#6a4"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -12808,7 +13814,9 @@ limitations under the License.
<th class="cap color-platform format-platform" style="color:#d77">Apache
Flink</th>
- <th class="cap color-platform format-platform" style="color:#d77">Apache
Spark</th>
+ <th class="cap color-platform format-platform" style="color:#d77">Apache
Spark (RDD/DStream based)</th>
+
+ <th class="cap color-platform format-platform" style="color:#d77">Apache
Spark Structured Streaming (Dataset based)</th>
<th class="cap color-platform format-platform" style="color:#d77">Apache
Apex</th>
@@ -12905,6 +13913,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#faa;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>Partially: fully supported in batch mode</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#d77;border-color:#b55"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -13126,6 +14152,24 @@ limitations under the License.
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>No</b></center><br />
+</td>
+
+
+
<td width="25%" class="cap"
style="background-color:#d77;border-color:#b55"><!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -13396,6 +14440,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
+<center><b>No: pending model support</b></center><br />
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#b55"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
<center><b>No</b></center><br />
</td>
@@ -13514,7 +14576,9 @@ limitations under the License.
<th class="cap color-platform format-platform" style="color:#bbb">Apache
Flink</th>
- <th class="cap color-platform format-platform" style="color:#bbb">Apache
Spark</th>
+ <th class="cap color-platform format-platform" style="color:#bbb">Apache
Spark (RDD/DStream based)</th>
+
+ <th class="cap color-platform format-platform" style="color:#bbb">Apache
Spark Structured Streaming (Dataset based)</th>
<th class="cap color-platform format-platform" style="color:#bbb">Apache
Apex</th>
@@ -13735,6 +14799,24 @@ limitations under the License.
<center><b>: </b></center><br />
</td>
+
+
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<center><b>: </b></center><br />
+</td>
+
</tr>
<tr class="cap">
@@ -13827,6 +14909,24 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
+<center><b>No: </b></center><br />not implemented
+</td>
+
+
+
+ <td width="25%" class="cap"
style="background-color:#ddd;border-color:#aaa"><!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
<center><b>: </b></center><br />
</td>
diff --git a/website/generated-content/documentation/runners/spark/index.html
b/website/generated-content/documentation/runners/spark/index.html
index 3da651c..f26f634 100644
--- a/website/generated-content/documentation/runners/spark/index.html
+++ b/website/generated-content/documentation/runners/spark/index.html
@@ -201,7 +201,8 @@
<ul class="nav">
- <li><a href="#portability">Portability</a></li>
+ <li><a href="#three-flavors-of-the-spark-runner">Three flavors of the Spark
runner</a></li>
+ <li><a href="#which-runner-to-use-portable-or-non-portable-runner">Which
runner to use: portable or non portable runner?</a></li>
<li><a href="#spark-runner-prerequisites-and-setup">Spark Runner
prerequisites and setup</a>
<ul>
<li><a href="#deploying-spark-with-your-application">Deploying Spark
with your application</a></li>
@@ -253,44 +254,50 @@ The Spark Runner can execute Spark pipelines just like a
native Spark applicatio
<p>The <a href="/documentation/runners/capability-matrix/">Beam Capability
Matrix</a> documents the currently supported capabilities of the Spark
Runner.</p>
-<p><em><strong>Note:</strong></em> <em>support for the Beam Model in streaming
is currently experimental, follow-up in the <a
href="/get-started/support/">mailing list</a> for status updates.</em></p>
-
-<h2 id="portability">Portability</h2>
-
-<p>The Spark runner comes in two flavors:</p>
+<h2 id="three-flavors-of-the-spark-runner">Three flavors of the Spark
runner</h2>
+<p>The Spark runner comes in three flavors:</p>
<ol>
- <li>A <em>legacy Runner</em> which supports only Java (and other JVM-based
languages)</li>
+ <li>A <em>legacy Runner</em> which supports only Java (and other JVM-based
languages) and that is based on Spark RDD/DStream</li>
+ <li>An <em>Structured Streaming Spark Runner</em> which supports only Java
(and other JVM-based languages) and that is based on Spark Datasets and the <a
href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html";>Apache
Spark Structured Streaming</a> framework.
+ <blockquote>
+ <p><strong>Note:</strong> It is still experimental, its coverage of the
Beam model is partial. As for now it only supports batch mode.</p>
+ </blockquote>
+ </li>
<li>A <em>portable Runner</em> which supports Java, Python, and Go</li>
</ol>
+<p>This guide is split into two parts to document the non-portable and
+the portable functionality of the Spark Runner. Please use the switcher below
to
+select the appropriate Runner:</p>
+
+<h2 id="which-runner-to-use-portable-or-non-portable-runner">Which runner to
use: portable or non portable runner?</h2>
+
<p>Beam and its Runners originally only supported JVM-based languages
(e.g. Java/Scala/Kotlin). Python and Go SDKs were added later on. The
architecture of the Runners had to be changed significantly to support
executing
pipelines written in other languages.</p>
-<p>If your applications only use Java, then you should currently go with the
legacy
-Runner. If you want to run Python or Go pipelines with Beam on Spark, you need
to use
+<p>If your applications only use Java, then you should currently go with one
of the java based runners.
+If you want to run Python or Go pipelines with Beam on Spark, you need to use
the portable Runner. For more information on portability, please visit the
<a href="/roadmap/portability/">Portability page</a>.</p>
-<p>This guide is split into two parts to document the legacy and
-the portable functionality of the Spark Runner. Please use the switcher below
to
-select the appropriate Runner:</p>
-
<nav class="language-switcher">
<strong>Adapt for:</strong>
<ul>
- <li data-type="language-java">Legacy (Java)</li>
+ <li data-type="language-java">Non portable (Java)</li>
<li data-type="language-py">Portable (Java/Python/Go)</li>
</ul>
</nav>
<h2 id="spark-runner-prerequisites-and-setup">Spark Runner prerequisites and
setup</h2>
-<p>The Spark runner currently supports Spark’s 2.x branch, and more
specifically any version greater than 2.2.0.</p>
+<p>The Spark runner currently supports Spark’s 2.x branch, and more
specifically any version greater than 2.4.0.</p>
-<p><span class="language-java">You can add a dependency on the latest version
of the Spark runner by adding to your pom.xml the following:</span></p>
+<p><span class="language-java">
+You can add a dependency on the latest version of the Spark runner by adding
to your pom.xml the following:
+</span></p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="o"><</span><span
class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">beam</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
@@ -301,22 +308,26 @@ select the appropriate Runner:</p>
<h3 id="deploying-spark-with-your-application">Deploying Spark with your
application</h3>
-<p><span class="language-java">In some cases, such as running in local
mode/Standalone, your (self-contained) application would be required to pack
Spark by explicitly adding the following dependencies in your
pom.xml:</span></p>
+<p><span class="language-java">
+In some cases, such as running in local mode/Standalone, your (self-contained)
application would be required to pack Spark by explicitly adding the following
dependencies in your pom.xml:
+</span></p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="o"><</span><span
class="n">dependency</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">spark</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
- <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">spark</span><span class="o">-</span><span
class="n">core_2</span><span class="o">.</span><span class="mi">10</span><span
class="o"></</span><span class="n">artifactId</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">spark</span><span class="o">-</span><span
class="n">core_2</span><span class="o">.</span><span class="mi">11</span><span
class="o"></</span><span class="n">artifactId</span><span
class="o">></span>
<span class="o"><</span><span class="n">version</span><span
class="o">></span><span class="err">$</span><span class="o">{</span><span
class="n">spark</span><span class="o">.</span><span
class="na">version</span><span class="o">}</</span><span
class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span
class="o">></span>
<span class="o"><</span><span class="n">dependency</span><span
class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">spark</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
- <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">spark</span><span class="o">-</span><span
class="n">streaming_2</span><span class="o">.</span><span
class="mi">10</span><span class="o"></</span><span
class="n">artifactId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">spark</span><span class="o">-</span><span
class="n">streaming_2</span><span class="o">.</span><span
class="mi">11</span><span class="o"></</span><span
class="n">artifactId</span><span class="o">></span>
<span class="o"><</span><span class="n">version</span><span
class="o">></span><span class="err">$</span><span class="o">{</span><span
class="n">spark</span><span class="o">.</span><span
class="na">version</span><span class="o">}</</span><span
class="n">version</span><span class="o">></span>
<span class="o"></</span><span class="n">dependency</span><span
class="o">></span>
</code></pre></div></div>
-<p><span class="language-java">And shading the application jar using the maven
shade plugin:</span></p>
+<p><span class="language-java">
+And shading the application jar using the maven shade plugin:
+</span></p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="o"><</span><span
class="n">plugin</span><span class="o">></span>
<span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">maven</span><span class="o">.</span><span
class="na">plugins</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
@@ -353,17 +364,30 @@ select the appropriate Runner:</p>
<span class="o"></</span><span class="n">plugin</span><span
class="o">></span>
</code></pre></div></div>
-<p><span class="language-java">After running <code>mvn package</code>, run
<code>ls target</code> and you should see (assuming your artifactId is <code
class="highlighter-rouge">beam-examples</code> and the version is <code
class="highlighter-rouge">1.0.0</code>):</span></p>
+<p><span class="language-java">
+After running <code>mvn package</code>, run <code>ls target</code> and you
should see (assuming your artifactId is <code
class="highlighter-rouge">beam-examples</code> and the version is <code
class="highlighter-rouge">1.0.0</code>):
+</span></p>
+
+<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="n">beam</span><span
class="o">-</span><span class="n">examples</span><span class="o">-</span><span
class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span
class="o">-</span><span class="n">shaded</span><span class="o">.</span><span
class="na">jar</span>
+</code></pre></div></div>
+
+<p><span class="language-java">
+To run against a Standalone cluster simply run:
+</span></p>
-<p><code class="language-java">
-beam-examples-1.0.0-shaded.jar
-</code></p>
+<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
+</span></p>
-<p><span class="language-java">To run against a Standalone cluster simply
run:</span></p>
+<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="n">spark</span><span
class="o">-</span><span class="n">submit</span> <span class="o">--</span><span
class="kd">class</span> <span class="nc">com</span><span
class="o">.</span><span class="na">beam</span><span class="o">.</span><span
class="na">examples</span><span class="o">.</span><span
class="na">BeamPipeline</span> <span class="o">--</span><span
class="n">master</span> <span c [...]
+</code></pre></div></div>
-<p><code class="language-java">
-spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT
target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
-</code></p>
+<p><span class="language-java">
+<br /><b>For Structured Streaming based runner:</b><br />
+</span></p>
+
+<div class="language-java highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="n">spark</span><span
class="o">-</span><span class="n">submit</span> <span class="o">--</span><span
class="kd">class</span> <span class="nc">com</span><span
class="o">.</span><span class="na">beam</span><span class="o">.</span><span
class="na">examples</span><span class="o">.</span><span
class="na">BeamPipeline</span> <span class="o">--</span><span
class="n">master</span> <span c [...]
+</code></pre></div></div>
<p><span class="language-py">
You will need Docker to be installed in your execution environment. To develop
@@ -381,8 +405,7 @@ download it on the <a
href="/get-started/downloads/">Downloads page</a>. In the
available.
</span></p>
-<p><span class="language-py">1. Start the JobService endpoint: <code
class="highlighter-rouge">./gradlew :runners:spark:job-server:runShadow</code>
-</span></p>
+<p><span class="language-py">1. Start the JobService endpoint: <code
class="highlighter-rouge">./gradlew
:runners:spark:job-server:runShadow</code></span></p>
<p><span class="language-py">
The JobService is the central instance where you submit your Beam pipeline.
@@ -391,8 +414,7 @@ job. To execute the job on a Spark cluster, the Beam
JobService needs to be
provided with the Spark master address.
</span></p>
-<p><span class="language-py">2. Submit the Python pipeline to the above
endpoint by using the <code class="highlighter-rouge">PortableRunner</code>,
<code class="highlighter-rouge">job_endpoint</code> set to <code
class="highlighter-rouge">localhost:8099</code> (this is the default address of
the JobService), and <code class="highlighter-rouge">environment_type</code>
set to <code class="highlighter-rouge">LOOPBACK</code>. For example:
-</span></p>
+<p><span class="language-py">2. Submit the Python pipeline to the above
endpoint by using the <code class="highlighter-rouge">PortableRunner</code>,
<code class="highlighter-rouge">job_endpoint</code> set to <code
class="highlighter-rouge">localhost:8099</code> (this is the default address of
the JobService), and <code class="highlighter-rouge">environment_type</code>
set to <code class="highlighter-rouge">LOOPBACK</code>. For example:</span></p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="kn">import</span> <span
class="nn">apache_beam</span> <span class="k">as</span> <span
class="n">beam</span>
<span class="kn">from</span> <span
class="nn">apache_beam.options.pipeline_options</span> <span
class="kn">import</span> <span class="n">PipelineOptions</span>
@@ -411,16 +433,13 @@ provided with the Spark master address.
<p>Deploying your Beam pipeline on a cluster that already has a Spark
deployment (Spark classes are available in container classpath) does not
require any additional dependencies.
For more details on the different deployment modes see: <a
href="http://spark.apache.org/docs/latest/spark-standalone.html";>Standalone</a>,
<a href="http://spark.apache.org/docs/latest/running-on-yarn.html";>YARN</a>,
or <a
href="http://spark.apache.org/docs/latest/running-on-mesos.html";>Mesos</a>.</p>
-<p><span class="language-py">1. Start a Spark cluster which exposes the master
on port 7077 by default.
-</span></p>
+<p><span class="language-py">1. Start a Spark cluster which exposes the master
on port 7077 by default.</span></p>
-<p><span class="language-py">2. Start JobService that will connect with the
Spark master: <code class="highlighter-rouge">./gradlew
:runners:spark:job-server:runShadow
-PsparkMasterUrl=spark://localhost:7077</code>.
-</span></p>
+<p><span class="language-py">2. Start JobService that will connect with the
Spark master: <code class="highlighter-rouge">./gradlew
:runners:spark:job-server:runShadow
-PsparkMasterUrl=spark://localhost:7077</code>.</span></p>
<p><span class="language-py">3. Submit the pipeline as above.
Note however that <code
class="highlighter-rouge">environment_type=LOOPBACK</code> is only intended for
local testing.
-See <a href="/roadmap/portability/#sdk-harness-config">here</a> for details.
-</span></p>
+See <a href="/roadmap/portability/#sdk-harness-config">here</a> for
details.</span></p>
<p><span class="language-py">
(Note that, depending on your cluster setup, you may need to change the <code
class="highlighter-rouge">environment_type</code> option.
@@ -431,6 +450,10 @@ See <a
href="/roadmap/portability/#sdk-harness-config">here</a> for details.)
<p>When executing your pipeline with the Spark Runner, you should consider the
following pipeline options.</p>
+<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
+</span></p>
+
<table class="language-java table table-bordered">
<tr>
<th>Field</th>
@@ -469,6 +492,53 @@ See <a
href="/roadmap/portability/#sdk-harness-config">here</a> for details.)
</tr>
</table>
+<p><span class="language-java">
+<br /><b>For Structured Streaming based runner:</b><br />
+</span></p>
+
+<table class="language-java table table-bordered">
+<tr>
+ <th>Field</th>
+ <th>Description</th>
+ <th>Default Value</th>
+</tr>
+<tr>
+ <td><code>runner</code></td>
+ <td>The pipeline runner to use. This option allows you to determine the
pipeline runner at runtime.</td>
+ <td>Set to <code>SparkStructuredStreamingRunner</code> to run using Spark
Structured Streaming.</td>
+</tr>
+<tr>
+ <td><code>sparkMaster</code></td>
+ <td>The url of the Spark Master. This is the equivalent of setting
<code>SparkConf#setMaster(String)</code> and can either be
<code>local[x]</code> to run local with x cores, <code>spark://host:port</code>
to connect to a Spark Standalone cluster, <code>mesos://host:port</code> to
connect to a Mesos cluster, or <code>yarn</code> to connect to a yarn
cluster.</td>
+ <td><code>local[4]</code></td>
+</tr>
+<tr>
+ <td><code>testMode</code></td>
+ <td>Enable test mode that gives useful debugging information: catalyst
execution plans and Beam DAG printing</td>
+ <td>false</td>
+</tr>
+<tr>
+ <td><code>enableSparkMetricSinks</code></td>
+ <td>Enable reporting metrics to Spark's metrics Sinks.</td>
+ <td>true</td>
+</tr>
+<tr>
+ <td><code>checkpointDir</code></td>
+ <td>A checkpoint directory for streaming resilience, ignored in batch. For
durability, a reliable filesystem such as HDFS/S3/GS is necessary.</td>
+ <td>local dir in /tmp</td>
+</tr>
+<tr>
+ <td><code>filesToStage</code></td>
+ <td>Jar-Files to send to all workers and put on the classpath.</td>
+ <td>all files from the classpath</td>
+</tr>
+<tr>
+ <td><code>EnableSparkMetricSinks</code></td>
+ <td>Enable/disable sending aggregator values to Spark's metric sinks</td>
+ <td>true</td>
+</tr>
+</table>
+
<table class="language-py table table-bordered">
<tr>
<th>Field</th>
@@ -509,17 +579,31 @@ Spark provides a <a
href="http://spark.apache.org/docs/latest/monitoring.html#me
<h3 id="streaming-execution">Streaming Execution</h3>
<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
If your pipeline uses an <code>UnboundedSource</code> the Spark Runner will
automatically set streaming mode. Forcing streaming mode is mostly used for
testing and is not recommended.
-</span>
-<span class="language-py">Streaming is not yet supported on the Spark portable
runner.</span></p>
+<br />
+<br /><b>For Structured Streaming based runner:</b><br />
+Streaming mode is not implemented yet in the Spark Structured Streaming runner.
+</span></p>
+
+<p><span class="language-py">
+Streaming is not yet supported on the Spark portable runner.
+</span></p>
<h3 id="using-a-provided-sparkcontext-and-streaminglisteners">Using a provided
SparkContext and StreamingListeners</h3>
<p><span class="language-java">
+<br /><b>For RDD/DStream based runner:</b><br />
If you would like to execute your Spark job with a provided
<code>SparkContext</code>, such as when using the <a
href="https://github.com/spark-jobserver/spark-jobserver";>spark-jobserver</a>,
or use <code>StreamingListeners</code>, you can’t use
<code>SparkPipelineOptions</code> (the context or a listener cannot be passed
as a command-line argument anyway).
Instead, you should use <code>SparkContextOptions</code> which can only be
used programmatically and is not a common <code>PipelineOptions</code>
implementation.
-</span>
-<span class="language-py">Provided SparkContext and StreamingListeners are not
supported on the Spark portable runner.</span></p>
+<br />
+<br /><b>For Structured Streaming based runner:</b><br />
+Provided SparkSession and StreamingListeners are not supported on the Spark
Structured Streaming runner
+</span></p>
+
+<p><span class="language-py">
+Provided SparkContext and StreamingListeners are not supported on the Spark
portable runner.
+</span></p>
</div>
</div>
