slinkydeveloper commented on a change in pull request #18353:
URL: https://github.com/apache/flink/pull/18353#discussion_r789423489
##########
File path: docs/content/docs/dev/configuration/overview.md
##########
@@ -52,46 +52,36 @@ In Maven syntax, it would look like:
{{< tabs "a49d57a4-27ee-4dd3-a2b8-a673b99b011e" >}}
{{< tab "Java" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
+
+{{< artifact flink-streaming-java withProvidedScope >}}
+
{{< /tab >}}
{{< tab "Scala" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< /tabs >}}
-**Important:** Note that all these dependencies have their scope set to
*provided*. This means that
-they are needed to compile against, but that they should not be packaged into
the project's resulting
-application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
-becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
-is that the Flink core dependencies that are added to the application's JAR
file clash with some of
-your own dependency versions (which is normally avoided through inverted
classloading).
+{{< artifact flink-streaming-scala withScalaVersion withProvidedScope >}}
-**Note on IntelliJ:** To make the applications run within IntelliJ IDEA, it is
necessary to tick the
-`Include dependencies with "Provided" scope` box in the run configuration. If
this option is not available
-(possibly due to using an older IntelliJ IDEA version), then a workaround is
to create a test that
-calls the application's `main()` method.
+{{< /tab >}}
+{{< /tabs >}}
Review comment:
Please remove these tabs and replace them with simple tabs for
maven/gradle/sbt conf like here:
https://github.com/slinkydeveloper/flink/commit/5d49dd7a0c0b0b824ed72942136a1857aaea91b9#diff-0bf4db953b94c9b897e098765f0ecf359afb3954363dd8c29574dbe3548c7d01R50
Telling me the syntax for the maven dependencies is not really useful here.
##########
File path: docs/content/docs/dev/table/sourcesSinks.md
##########
@@ -106,6 +106,41 @@ that the planner can handle.
{{< top >}}
+
+Project Configuration
+---------------------
+
+If you want to implement a custom format, the following dependency is usually
sufficient and can be
+used for JAR files for the SQL Client:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>{{< version >}}</version>
+ <scope>provided</scope>
+</dependency>
+```
+
+If you want to develop a connector that needs to bridge with DataStream APIs
(i.e. if you want to adapt
+a DataStream connector to the Table API), you need to add this dependency:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>{{< version >}}</version>
+ <scope>provided</scope>
+</dependency>
+```
Review comment:
Use the artifact docgen tag
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
+
+## Adding Connector Dependencies
+
+As an example, you can add the Kafka connector as a dependency like this
(Maven syntax):
+
+{{< artifact flink-connector-kafka >}}
+
+We recommend packaging the application code and all its required dependencies
into one *JAR-with-dependencies*
+which we refer to as the *application JAR*. The application JAR can be
submitted to an already running
+Flink cluster, or added to a Flink application container image.
+
+Projects created from the `Java Project Template`, the `Scala Project
Template`, or Gradle are configured
+to automatically include the application dependencies into the application JAR
when you run `mvn clean package`.
+For projects that are not set up from those templates, we recommend adding the
Maven Shade Plugin to
+build the application jar with all required dependencies.
+
+**Important:** For Maven (and other build tools) to correctly package the
dependencies into the application jar,
+these application dependencies must be specified in scope *compile* (unlike
the core dependencies, which
+must be specified in scope *provided*).
+
+## Packaging Dependencies
+
+In the Maven Repository, you will find connectors named
"flink-connector-<NAME>" and
+"flink-sql-connector-<NAME>". The former are thin JARs while the latter are
uber JARs.
+
+In order to use the uber JARs, you can shade them in the uber JAR of your
application, or you can add
+them to the `/lib` folder of the distribution (i.e. SQL client).
+
+[ EXPLAIN PROS and CONS ]
+
+In order to create an uber JAR to run the job, do this:
+
+[ FILL IN ]
+
+**Note:** You do not need to shade Flink API dependencies. You only need to do
this for connectors,
+formats and third-party dependencies.
Review comment:
For Maven specifically, what you have in `advanced.md > Template for
building a JAR with Dependencies` is ok to start with (see
https://github.com/slinkydeveloper/flink/commit/41ea33a191bc7e8459c3ca6372cdeb675e9f8552)
##########
File path: docs/content/docs/dev/configuration/overview.md
##########
@@ -52,46 +52,36 @@ In Maven syntax, it would look like:
{{< tabs "a49d57a4-27ee-4dd3-a2b8-a673b99b011e" >}}
{{< tab "Java" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
+
+{{< artifact flink-streaming-java withProvidedScope >}}
+
{{< /tab >}}
{{< tab "Scala" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< /tabs >}}
-**Important:** Note that all these dependencies have their scope set to
*provided*. This means that
-they are needed to compile against, but that they should not be packaged into
the project's resulting
-application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
-becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
-is that the Flink core dependencies that are added to the application's JAR
file clash with some of
-your own dependency versions (which is normally avoided through inverted
classloading).
+{{< artifact flink-streaming-scala withScalaVersion withProvidedScope >}}
-**Note on IntelliJ:** To make the applications run within IntelliJ IDEA, it is
necessary to tick the
-`Include dependencies with "Provided" scope` box in the run configuration. If
this option is not available
-(possibly due to using an older IntelliJ IDEA version), then a workaround is
to create a test that
-calls the application's `main()` method.
+{{< /tab >}}
+{{< /tabs >}}
## Which dependencies do you need?
+Different APIs will require different dependencies.
+
| APIs you want to use | Dependency you need to add |
|-----------------------------------|-------------------------------|
| DataStream | flink-streaming-java |
-| DataStream with Scala | flink-streaming-scala |
+| DataStream with Scala | flink-streaming-scala{{< scala_version
>}} |
| Table API | flink-table-api-java |
-| Table API with Scala | flink-table-api-scala |
+| Table API with Scala | flink-table-api-scala{{< scala_version
>}} |
| Table API + DataStream | flink-table-api-java-bridge |
-| Table API + DataStream with Scala | flink-table-api-scala-bridge |
+| Table API + DataStream with Scala | flink-table-api-scala-bridge{{<
scala_version >}} |
+
+You can use [Maven]({{< ref "docs/dev/configuration/maven" >}}), [Gradle]({{<
ref "docs/dev/configuration/gradle" >}}),
+or [sbt]({{< ref "docs/dev/configuration/sbt" >}}) to configure your project
and add these dependencies.
+
+**Important:** Note that all these dependencies should have their scope set to
*provided*. This means that
+they are needed to compile against, but that they should not be packaged into
the project's resulting
+application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
+becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
+is that the Flink core dependencies that are added to the application's JAR
file clash with some of
+your own dependency versions (which is normally avoided through inverted
classloading).
Review comment:
I would move that to the maven/gradle/sbt specific pages
##########
File path: docs/content/docs/dev/table/sourcesSinks.md
##########
@@ -106,6 +106,41 @@ that the planner can handle.
{{< top >}}
+
+Project Configuration
+---------------------
+
+If you want to implement a custom format, the following dependency is usually
sufficient and can be
+used for JAR files for the SQL Client:
Review comment:
Remove the wording after "is sufficient"
##########
File path: docs/content/docs/dev/table/sourcesSinks.md
##########
@@ -106,6 +106,41 @@ that the planner can handle.
{{< top >}}
+
+Project Configuration
+---------------------
+
+If you want to implement a custom format, the following dependency is usually
sufficient and can be
+used for JAR files for the SQL Client:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>{{< version >}}</version>
+ <scope>provided</scope>
+</dependency>
+```
Review comment:
Use the artifact docgen tag
##########
File path: docs/content/docs/dev/configuration/overview.md
##########
@@ -52,46 +52,36 @@ In Maven syntax, it would look like:
{{< tabs "a49d57a4-27ee-4dd3-a2b8-a673b99b011e" >}}
{{< tab "Java" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
+
+{{< artifact flink-streaming-java withProvidedScope >}}
+
{{< /tab >}}
{{< tab "Scala" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< /tabs >}}
-**Important:** Note that all these dependencies have their scope set to
*provided*. This means that
-they are needed to compile against, but that they should not be packaged into
the project's resulting
-application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
-becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
-is that the Flink core dependencies that are added to the application's JAR
file clash with some of
-your own dependency versions (which is normally avoided through inverted
classloading).
+{{< artifact flink-streaming-scala withScalaVersion withProvidedScope >}}
-**Note on IntelliJ:** To make the applications run within IntelliJ IDEA, it is
necessary to tick the
-`Include dependencies with "Provided" scope` box in the run configuration. If
this option is not available
-(possibly due to using an older IntelliJ IDEA version), then a workaround is
to create a test that
-calls the application's `main()` method.
+{{< /tab >}}
+{{< /tabs >}}
## Which dependencies do you need?
+Different APIs will require different dependencies.
Review comment:
Try to be less formal on this page? Like
> At this point you have a ready to play project configuration. Now,
depending on what you're trying to achieve, you're going to choose a
combination of our available APIs[...] This is a table of artifact names:
##########
File path: docs/content/docs/dev/table/data_stream_api.md
##########
@@ -412,60 +412,7 @@ also the [dedicated batch mode section
below](#batch-runtime-mode) for more insi
### Dependencies and Imports
-Projects that combine Table API with DataStream API need to add one of the
following bridging modules.
-They include transitive dependencies to `flink-table-api-java` or
`flink-table-api-scala` and the
-corresponding language-specific DataStream API module.
-
-{{< tabs "0d2da52a-ee43-4d06-afde-b165517c0617" >}}
-{{< tab "Java" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-java-bridge{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< tab "Scala" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-api-scala-bridge{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< /tabs >}}
-
-The following imports are required to declare common pipelines using either
the Java or Scala version
Review comment:
This code below is still valid, keep it.
##########
File path: docs/content/docs/dev/table/sourcesSinks.md
##########
@@ -106,6 +106,41 @@ that the planner can handle.
{{< top >}}
+
+Project Configuration
+---------------------
+
+If you want to implement a custom format, the following dependency is usually
sufficient and can be
Review comment:
"a custom connector or a custom format"
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
Review comment:
This wording is valid and correct but it's too specific to table,
generalize it for DataStream as well
##########
File path: docs/content/docs/dev/configuration/overview.md
##########
@@ -52,46 +52,36 @@ In Maven syntax, it would look like:
{{< tabs "a49d57a4-27ee-4dd3-a2b8-a673b99b011e" >}}
{{< tab "Java" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
+
+{{< artifact flink-streaming-java withProvidedScope >}}
+
{{< /tab >}}
{{< tab "Scala" >}}
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{< scala_version >}}</artifactId>
- <version>{{< version >}}</version>
- <scope>provided</scope>
-</dependency>
-```
-{{< /tab >}}
-{{< /tabs >}}
-**Important:** Note that all these dependencies have their scope set to
*provided*. This means that
-they are needed to compile against, but that they should not be packaged into
the project's resulting
-application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
-becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
-is that the Flink core dependencies that are added to the application's JAR
file clash with some of
-your own dependency versions (which is normally avoided through inverted
classloading).
+{{< artifact flink-streaming-scala withScalaVersion withProvidedScope >}}
-**Note on IntelliJ:** To make the applications run within IntelliJ IDEA, it is
necessary to tick the
-`Include dependencies with "Provided" scope` box in the run configuration. If
this option is not available
-(possibly due to using an older IntelliJ IDEA version), then a workaround is
to create a test that
-calls the application's `main()` method.
+{{< /tab >}}
+{{< /tabs >}}
## Which dependencies do you need?
+Different APIs will require different dependencies.
+
| APIs you want to use | Dependency you need to add |
|-----------------------------------|-------------------------------|
| DataStream | flink-streaming-java |
-| DataStream with Scala | flink-streaming-scala |
+| DataStream with Scala | flink-streaming-scala{{< scala_version
>}} |
| Table API | flink-table-api-java |
-| Table API with Scala | flink-table-api-scala |
+| Table API with Scala | flink-table-api-scala{{< scala_version
>}} |
| Table API + DataStream | flink-table-api-java-bridge |
-| Table API + DataStream with Scala | flink-table-api-scala-bridge |
+| Table API + DataStream with Scala | flink-table-api-scala-bridge{{<
scala_version >}} |
+
+You can use [Maven]({{< ref "docs/dev/configuration/maven" >}}), [Gradle]({{<
ref "docs/dev/configuration/gradle" >}}),
+or [sbt]({{< ref "docs/dev/configuration/sbt" >}}) to configure your project
and add these dependencies.
+
+**Important:** Note that all these dependencies should have their scope set to
*provided*. This means that
+they are needed to compile against, but that they should not be packaged into
the project's resulting
+application JAR file. If not set to *provided*, the best case scenario is that
the resulting JAR
+becomes excessively large, because it also contains all Flink core
dependencies. The worst case scenario
+is that the Flink core dependencies that are added to the application's JAR
file clash with some of
+your own dependency versions (which is normally avoided through inverted
classloading).
Review comment:
Missing a next steps section here linking to the other pages
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
+
+## Adding Connector Dependencies
+
+As an example, you can add the Kafka connector as a dependency like this
(Maven syntax):
+
+{{< artifact flink-connector-kafka >}}
+
+We recommend packaging the application code and all its required dependencies
into one *JAR-with-dependencies*
+which we refer to as the *application JAR*. The application JAR can be
submitted to an already running
+Flink cluster, or added to a Flink application container image.
+
+Projects created from the `Java Project Template`, the `Scala Project
Template`, or Gradle are configured
+to automatically include the application dependencies into the application JAR
when you run `mvn clean package`.
+For projects that are not set up from those templates, we recommend adding the
Maven Shade Plugin to
+build the application jar with all required dependencies.
+
+**Important:** For Maven (and other build tools) to correctly package the
dependencies into the application jar,
+these application dependencies must be specified in scope *compile* (unlike
the core dependencies, which
+must be specified in scope *provided*).
Review comment:
All of this is very build tool specific, please split this wording and
put it in the various maven/gradle pages, and then link them here
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
+
+## Adding Connector Dependencies
+
+As an example, you can add the Kafka connector as a dependency like this
(Maven syntax):
+
+{{< artifact flink-connector-kafka >}}
+
+We recommend packaging the application code and all its required dependencies
into one *JAR-with-dependencies*
+which we refer to as the *application JAR*. The application JAR can be
submitted to an already running
+Flink cluster, or added to a Flink application container image.
+
+Projects created from the `Java Project Template`, the `Scala Project
Template`, or Gradle are configured
+to automatically include the application dependencies into the application JAR
when you run `mvn clean package`.
+For projects that are not set up from those templates, we recommend adding the
Maven Shade Plugin to
+build the application jar with all required dependencies.
+
+**Important:** For Maven (and other build tools) to correctly package the
dependencies into the application jar,
+these application dependencies must be specified in scope *compile* (unlike
the core dependencies, which
+must be specified in scope *provided*).
+
+## Packaging Dependencies
+
+In the Maven Repository, you will find connectors named
"flink-connector-<NAME>" and
+"flink-sql-connector-<NAME>". The former are thin JARs while the latter are
uber JARs.
+
+In order to use the uber JARs, you can shade them in the uber JAR of your
application, or you can add
+them to the `/lib` folder of the distribution (i.e. SQL client).
+
+[ EXPLAIN PROS and CONS ]
+
+In order to create an uber JAR to run the job, do this:
+
+[ FILL IN ]
+
+**Note:** You do not need to shade Flink API dependencies. You only need to do
this for connectors,
+formats and third-party dependencies.
Review comment:
This should not be here, and should rather link to the build tool
specifics. Every build tool works differently for shading, so we need a shading
section both for Maven and Gradle (also for SBT, but I don't know how it works
for them so just leave it empty now for them), and then link it from here.
##########
File path: docs/content/docs/dev/table/sqlClient.md
##########
@@ -417,14 +417,17 @@ When execute queries or insert statements, please enter
the interactive mode or
### Dependencies
-The SQL Client does not require to setup a Java project using Maven or SBT.
Instead, you can pass the
+The SQL Client does not require setting up a Java project using Maven or sbt.
Instead, you can pass the
Review comment:
"Maven, Gradle or SBT"
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
+
+## Adding Connector Dependencies
+
+As an example, you can add the Kafka connector as a dependency like this
(Maven syntax):
+
+{{< artifact flink-connector-kafka >}}
+
+We recommend packaging the application code and all its required dependencies
into one *JAR-with-dependencies*
+which we refer to as the *application JAR*. The application JAR can be
submitted to an already running
+Flink cluster, or added to a Flink application container image.
+
+Projects created from the `Java Project Template`, the `Scala Project
Template`, or Gradle are configured
+to automatically include the application dependencies into the application JAR
when you run `mvn clean package`.
+For projects that are not set up from those templates, we recommend adding the
Maven Shade Plugin to
+build the application jar with all required dependencies.
+
+**Important:** For Maven (and other build tools) to correctly package the
dependencies into the application jar,
+these application dependencies must be specified in scope *compile* (unlike
the core dependencies, which
+must be specified in scope *provided*).
+
+## Packaging Dependencies
+
+In the Maven Repository, you will find connectors named
"flink-connector-<NAME>" and
+"flink-sql-connector-<NAME>". The former are thin JARs while the latter are
uber JARs.
+
+In order to use the uber JARs, you can shade them in the uber JAR of your
application, or you can add
+them to the `/lib` folder of the distribution (i.e. SQL client).
+
+[ EXPLAIN PROS and CONS ]
Review comment:
I start suggesting some wording:
> When loading uber jars directly in the distribution, you lock on a
specific version of the connector, simplifying the management of a shared
multi-job flink cluster, but as a downside each job developer doesn't have
control over the connector version. While when shading the connector jar,
either the thin or the fat one, in your job JAR, guarantees each job developer
more control over the connector version. In case of shading the thin JAR, the
job developer has even control over the transitive dependencies, as it can bump
them without bumping the connector (binary compatibility permitting).
@fapaul @twalthr wanna add some details here on why one should use thin vs
uber jars?
##########
File path: docs/content/docs/dev/configuration/connector.md
##########
@@ -0,0 +1,72 @@
+---
+title: "Dependencies: Connectors and Formats"
+weight: 5
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Dependencies: Connectors and Formats
+
+Flink can read from and write to various external systems via [connectors]({{<
ref "docs/connectors/table/overview" >}})
+and define the [format]({{< ref "docs/connectors/table/formats/overview" >}})
in which to store the
+data (i.e. mapping binary data onto table columns).
+
+The way that the information is serialized is represented in the external
system and that system needs
+to know how to read this data in a format that can be read by Flink. This is
done through format dependencies.
+
+Most applications need specific connectors to run. Flink provides a set of
table formats that can be
+used with table connectors (with the dependencies for both being fairly
unified). These are not part
+of Flink's core dependencies and must be added as dependencies to the
application.
+
+## Adding Connector Dependencies
+
+As an example, you can add the Kafka connector as a dependency like this
(Maven syntax):
+
+{{< artifact flink-connector-kafka >}}
+
+We recommend packaging the application code and all its required dependencies
into one *JAR-with-dependencies*
+which we refer to as the *application JAR*. The application JAR can be
submitted to an already running
+Flink cluster, or added to a Flink application container image.
+
+Projects created from the `Java Project Template`, the `Scala Project
Template`, or Gradle are configured
+to automatically include the application dependencies into the application JAR
when you run `mvn clean package`.
+For projects that are not set up from those templates, we recommend adding the
Maven Shade Plugin to
+build the application jar with all required dependencies.
+
+**Important:** For Maven (and other build tools) to correctly package the
dependencies into the application jar,
+these application dependencies must be specified in scope *compile* (unlike
the core dependencies, which
+must be specified in scope *provided*).
+
+## Packaging Dependencies
+
+In the Maven Repository, you will find connectors named
"flink-connector-<NAME>" and
Review comment:
> In the Maven Repository, you will find connectors
To
> On Maven Central we publish connectors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
