linton-galloway-bloom commented on issue #32944:
URL: https://github.com/apache/beam/issues/32944#issuecomment-2619222938
Hi @siddhanta-rath - I've managed to get it working with:
* Flink Docker image=flink:1.19.0-java11
* Beam SDK=2.62.0
* Java Version=11/Corretto
* Flink Operator=1.10.0
**pom configuration:**
``` xml
<properties>
<beam.version>2.62.0</beam.version>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
<maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-flink-1.19</artifactId>
<version>${beam.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${maven-exec-plugin.version}</version>
<configuration>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<profiles>
<profile>
<id>package</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<finalName>job</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>job</finalName>
<shadedArtifactAttached>false</shadedArtifactAttached>
<filters>
<filter>
<!-- Do not copy the signatures
in the META-INF folder.
Otherwise, this might cause
SecurityExceptions when using the JAR. -->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>some.path.WordCount</mainClass>
</transformer>
</transformers>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
```
**Operator configuration:**
```yaml
---
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
creationTimestamp: "..."
finalizers:
- flinkdeployments.flink.apache.org/finalizer
generation: 26
labels:
argocd.argoproj.io/instance: some-instance-name
name: some-name
namespace: some-namespace
spec:
flinkConfiguration:
kubernetes.operator.job.autoscaler.enabled: "true"
kubernetes.operator.job.autoscaler.metrics.window: 3m
kubernetes.operator.job.autoscaler.stabilization.interval: 1m
metrics.reporter.prom.factory.class:
org.apache.flink.metrics.prometheus.PrometheusReporterFactory
metrics.reporters: prom
pipeline.max-parallelism: "24"
taskmanager.numberOfTaskSlots: "2"
flinkVersion: v1_19
image: ghcr.io/some/image:tag
imagePullPolicy: Always
job:
args:
- --runner=FlinkRunner
- --attachedMode=false
- --output=file://opt/output.txt
entryClass: some-path.WordCount
jarURI: local:///opt/flink/lib/job.jar
parallelism: 2
state: running
upgradeMode: stateless
jobManager:
replicas: 1
resource:
cpu: 1
memory: 2048m
podTemplate:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: prom
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/instance: some-instance
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: some-name
app.kubernetes.io/version: 1.16.0
helm.sh/chart: some-helm-chart
spec:
containers:
- name: flink-main-container
ports:
- containerPort: 9249
name: prom
protocol: TCP
imagePullSecrets:
- name: regcred
serviceAccount: some-service-account
taskManager:
resource:
cpu: 1
memory: 2048m
```
**Dockerfile:**
```Dockerfile
FROM flink:1.19.0-java11
COPY target/lib /opt/flink/lib
COPY target/job.jar /opt/flink/lib
```
**Reference code:**
https://github.com/apache/beam/blob/489e75e6c652c2f103c21e301abd5e1b4e882a95/examples/java/src/main/java/org/apache/beam/examples/WordCount.java#L99
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
