Hi Salva,
I think you're almost there. Confusion is definitely not helped by the
ADDONS/ PROVIDED_ADDONS thingy – I think I tried to get too fancy with that
in the linked thread.
I think the only thing you have to do differently is to adjust the target
you are building/ deploying – instead of
`//src/main/scala/org/example:flink_app_deploy.jar`, your target with the
provided env applied is
`//src/main/scala/org/example:word_count_deploy.jar`. I've verified this in
the following ways:
1. Building and checking the JAR itself
* bazel build //src/main/scala/org/example:word_count_deploy.jar
* jar -tf bazel-bin/src/main/scala/org/example/word_count_deploy.jar | grep
flink
* Shows only the tools/flink/NoOp class
2. Running the word count jar locally, to ensure the main class is picked
up correctly:
./bazel-bin/src/main/scala/org/example/word_count
USAGE:
WordCount <hostname> <port>
3. Had fun with the Bazel query language[1], inspecting the difference in
the dependencies between the deploy env and the word_cound_deploy.jar:
bazel query 'filter("@maven//:org_apache_flink.*",
deps(//src/main/scala/org/example:word_count_deploy.jar) except
deps(//:default_flink_deploy_env))'
INFO: Empty results
Loading: 0 packages loaded
This is to say that there are no Flink dependencies in the deploy JAR that
are not accounted for in the deploy env.
So I think you're all good, but let me know if I've misunderstood! Or if
you find a better way of doing the provided deps – I'd be very interested!
Best,
Austin
[1]: https://docs.bazel.build/versions/master/query.htm
On Wed, May 12, 2021 at 10:28 AM Salva Alcántara <salcantara...@gmail.com>
wrote:
> Hi Austin,
>
> I followed your instructions and gave `rules_jvm_external` a try.
>
> Overall, I think I advanced a bit, but I'm not quite there yet. I have
> followed the link [1] given by Matthias, making the necessary changes to my
> repo:
>
> https://github.com/salvalcantara/bazel-flink-scala
>
> In particular, the relevant (bazel) BUILD file looks like this:
>
> ```
> package(default_visibility = ["//visibility:public"])
>
> load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library",
> "scala_test")
>
> filegroup(
> name = "scala-main-srcs",
> srcs = glob(["*.scala"]),
> )
>
> scala_library(
> name = "flink_app",
> srcs = [":scala-main-srcs"],
> deps = [
> "@maven//:org_apache_flink_flink_core",
> "@maven//:org_apache_flink_flink_clients_2_12",
> "@maven//:org_apache_flink_flink_scala_2_12",
> "@maven//:org_apache_flink_flink_streaming_scala_2_12",
> "@maven//:org_apache_flink_flink_streaming_java_2_12",
> ],
> )
>
> java_binary(
> name = "word_count",
> srcs = ["//tools/flink:noop"],
> deploy_env = ["//:default_flink_deploy_env"],
> main_class = "org.example.WordCount",
> deps = [
> ":flink_app",
> ],
> )
> ```
>
> The idea is to use `deploy_env` within `java_binary` for providing the
> flink
> dependencies. This causes those dependencies to get removed from the final
> fat jar that one gets by running:
>
> ```
> bazel build //src/main/scala/org/example:flink_app_deploy.jar
> ```
>
> The problem now is that the jar still includes the Scala library, which
> should also be dropped from the jar as it is part of the provided
> dependencies within the Flink cluster. I am reading this blog post in [2]
> without luck yet...
>
> Regards,
>
> Salva
>
> [1]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-anyone-have-an-example-of-Bazel-working-with-Flink-td35898.html
>
> [2]
>
> https://yishanhe.net/address-dependency-conflict-for-bazel-built-scala-spark/
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
