Abacn commented on code in PR #27805:
URL: https://github.com/apache/beam/pull/27805#discussion_r1289022034
##########
sdks/java/io/google-cloud-platform/expansion-service/build.gradle:
##########
@@ -34,6 +34,8 @@ dependencies {
permitUnusedDeclared project(":sdks:java:expansion-service") // BEAM-11761
implementation project(":sdks:java:io:google-cloud-platform")
permitUnusedDeclared project(":sdks:java:io:google-cloud-platform") //
BEAM-11761
+ implementation project(":sdks:java:io:kafka")
Review Comment:
gcp expansion service should not depend on kafka. Note that expansion
service jar is a shadowJar. All transient dependencies will be added.
Consider make the test suite depend on
project(`:sdks:java:io:expansion-service`) and invoke the io-expansion-service
in kafka test
##########
sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py:
##########
@@ -138,6 +143,44 @@ def test_kafkaio_null_key(self):
self.run_kafka_write(pipeline_creator)
self.run_kafka_read(pipeline_creator, None)
+ @pytest.mark.uses_gcp_java_expansion_service
Review Comment:
This in combine with the kafka dependency in gcp expansion service work in a
way that inject kafka dependency and tests into existing gcp_io_xlang test
suite. I would suggest think about a more clean way to do this.
e.g. consider another mark for kafka test. It does not use PTransforms in
:sdks:java:io:google-cloud-platform project
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -2522,6 +2522,8 @@ class BeamModulePlugin implements Plugin<Project> {
project.exec {
environment "EXPANSION_JAR", expansionJar
environment "EXPANSION_PORT", javaExpansionPort
+ //Use hardcoded kafka temporarily until hosted Kafka on Google is
available. TODO: johnjcasey
+ environment "BOOTSTRAP_SERVER",
'theotherjohn-nokill-kafka-c-m:9092'
Review Comment:
For README, note that we already have
https://github.com/apache/beam/tree/master/.test-infra/kafka and one can use it
to deploy kafka cluster, k8s based.
##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -2522,6 +2522,8 @@ class BeamModulePlugin implements Plugin<Project> {
project.exec {
environment "EXPANSION_JAR", expansionJar
environment "EXPANSION_PORT", javaExpansionPort
+ //Use hardcoded kafka temporarily until hosted Kafka on Google is
available. TODO: johnjcasey
+ environment "BOOTSTRAP_SERVER",
'theotherjohn-nokill-kafka-c-m:9092'
Review Comment:
honestly not comfortable add a private domain to BeamModulePlugin (even with
sort of PII). Can we configure it some more specific path, like build.gradle or
tox file that invokes this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
