[
https://issues.apache.org/jira/browse/BEAM-8345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944894#comment-16944894
]
Valentyn Tymofieiev commented on BEAM-8345:
-------------------------------------------
I think we are missing Flink/Spark runner validations in automated scripts.
When interacting with Flink/Spark, we can run Beam SDK harness in a container
or local process. Container execution can also help validate the container
images we release.
I manually tried validating Flink/Spark using commands at the end of this
message. However that sequence of commands may not be very robust since the
output of wordcount is available only inside a container
(https://issues.apache.org/jira/browse/BEAM-7857), and the container seems to
be garbage-collected shortly after execution. Also, we need to pay attention to
which version of Flink we are using. I tried 1.5, but we may need to test
against all versions we support.
Related issue that we may need to address:
https://issues.apache.org/jira/browse/BEAM-7858.
{noformat}
git checkout tags/v2.16.0-RC1
./gradlew :runners:spark:job-server:runShadow or ./gradlew
:runners:flink:1.5:job-server:runShadow
In a separate terminal:
docker pull apachebeam/python3.5_sdk:2.16.0_rc1
docker tag apachebeam/python3.5_sdk:2.16.0_rc1 apachebeam/python3.5_sdk:2.16.0
./gradlew :sdks:python:test-suites:portable:py35:portableWordCountBatch
-PjobEndpoint=localhost:8099 -PenvirionmentType=DOCKER
As soon as that is done:
docker ps -a | grep apachebeam
Note the container ID of a running container. It stays around only for a minute
or so, and gets garbage-collected...
docker exec 866fb8932207 /bin/bash -c 'cat /tmp/py-wordcount*'
<wordcount output is printed>
{noformat}
> Add missing validations to run_rc_validation.sh
> -----------------------------------------------
>
> Key: BEAM-8345
> URL: https://issues.apache.org/jira/browse/BEAM-8345
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Mark Liu
> Assignee: Mark Liu
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
