spuru9 commented on code in PR #56:
URL:
https://github.com/apache/flink-connector-cassandra/pull/56#discussion_r4633069720
##########
flink-connector-cassandra/pom.xml:
##########
@@ -38,16 +38,15 @@ under the License.
<!-- Allow users to pass custom connector versions -->
<properties>
<scala.binary.version>2.12</scala.binary.version>
- <scala-library.version>2.12.7</scala-library.version>
<slf4j.version>1.7.36</slf4j.version>
<junit5.version>5.8.1</junit5.version>
- <assertj.version>3.23.1</assertj.version>
+ <assertj.version>3.27.3</assertj.version>
<archunit.version>1.0.0</archunit.version>
<!-- For dependency convergence we exclude testContainers
commons-compress transitive dependency and keep only flink's.
Don't downgrade testcontainers otherwise the wrong
commons-compress transitive dependency will lead to NoClassDefFound-->
<testcontainers.version>1.21.4</testcontainers.version>
- <mockito.version>3.4.6</mockito.version>
- <byte-buddy.version>1.12.10</byte-buddy.version>
+ <mockito.version>5.19.0</mockito.version>
Review Comment:
Mockito 5.19 needs objenesis 3.3 and Flink 2.3's Kryo needs 3.4, but the
`objenesis` 2.6 convergence pin further down still forces the old one. Could we
bump it to 3.4 to match Flink? `junit5.version` 5.8.1 is also well behind Flink
2.3's 5.11.4.
##########
pom.xml:
##########
@@ -168,4 +168,28 @@ under the License.
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>java21</id>
+ <activation>
+ <jdk>[21,)</jdk>
+ </activation>
+ <build>
+ <pluginManagement>
+ <plugins>
+ <plugin>
+
<groupId>com.diffplug.spotless</groupId>
+
<artifactId>spotless-maven-plugin</artifactId>
+ <configuration>
+ <!--
google-java-format 1.7 from the parent does not run on Java 21.
+ Same as
the java21 profile in flink-connector-parent 2.0.0. -->
+
<skip>true</skip>
+ </configuration>
+ </plugin>
+ </plugins>
+ </pluginManagement>
+ </build>
+ </profile>
+ </profiles>
</project>
Review Comment:
Not a blocker, but kafka, jdbc, es and mongodb are already on
`flink-connector-parent` 2.0.0. It ships this exact profile, and 1.1.0 still
defaults to a Java 8 target. Any reason not to bump the parent instead of
copying the profile? A follow-up is fine too.
##########
flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java:
##########
@@ -456,6 +449,41 @@ void testCassandraCommitter() throws Exception {
cc1.close();
}
+ @Test
+ void testCassandraTupleWriteAheadSinkInJob() throws Exception {
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.enableCheckpointing(100);
+ addWriteAheadSink(env);
+ env.execute();
+
+ ResultSet rs =
+ cassandraTestEnvironment.executeRequestWithTimeout(
+ injectTableName(SELECT_DATA_QUERY));
+ assertThat(rs.all()).hasSameSizeAs(collection);
+ }
+
+ @Test
+ void testCassandraTupleWriteAheadSinkRequiresCheckpointing() throws
Exception {
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ addWriteAheadSink(env);
+
+ assertThatThrownBy(env::execute)
+ .satisfies(
+ anyCauseMatches(
+ IllegalStateException.class,
+ "The write-ahead log requires checkpointing to
be enabled."));
+ }
+
+ private void addWriteAheadSink(StreamExecutionEnvironment env) throws
Exception {
+ CassandraSink.addSink(env.fromData(collection))
+ .setQuery(injectTableName(INSERT_DATA_QUERY))
+
.setClusterBuilder(cassandraTestEnvironment.getBuilderForWriting())
+ .enableWriteAheadLog()
+ .build();
+ }
Review Comment:
`CassandraRowWriteAheadSink` got the same checkpointing check but has no
test at all. Could this helper take the input stream, so the two job-level
tests also run with `rowCollection`?
##########
flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java:
##########
@@ -682,114 +668,4 @@ void testCassandraBatchRowFormat() throws Exception {
List<com.datastax.driver.core.Row> rows = rs.all();
assertThat(rows).hasSameSizeAs(rowCollection);
}
-
- @Test
- void testCassandraScalaTupleAtLeastOnceSinkBuilderDetection() throws
Exception {
- Class<scala.Tuple1<String>> c =
- (Class<scala.Tuple1<String>>) new
scala.Tuple1<>("hello").getClass();
- Seq<TypeInformation<?>> typeInfos =
- JavaConverters.asScalaBufferConverter(
- Collections.<TypeInformation<?>>singletonList(
- BasicTypeInfo.STRING_TYPE_INFO))
- .asScala();
- Seq<String> fieldNames =
-
JavaConverters.asScalaBufferConverter(Collections.singletonList("_1")).asScala();
-
- CaseClassTypeInfo<scala.Tuple1<String>> typeInfo =
- new CaseClassTypeInfo<scala.Tuple1<String>>(c, null,
typeInfos, fieldNames) {
- @Override
- public TypeSerializer<scala.Tuple1<String>>
createSerializer(
- ExecutionConfig config) {
- return null;
- }
- };
-
- StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
- DataStream<scala.Tuple1<String>> input =
- env.fromElements(new
scala.Tuple1<>("hello")).returns(typeInfo);
-
- CassandraSink.CassandraSinkBuilder<scala.Tuple1<String>> sinkBuilder =
- CassandraSink.addSink(input);
-
assertThat(sinkBuilder).isInstanceOf(CassandraSink.CassandraScalaProductSinkBuilder.class);
- }
-
- @Test
- void testCassandraScalaTupleAtLeastSink() throws Exception {
- CassandraScalaProductSink<scala.Tuple3<String, Integer, Integer>> sink
=
- new CassandraScalaProductSink<>(
- injectTableName(INSERT_DATA_QUERY),
- cassandraTestEnvironment.getBuilderForWriting());
-
- List<scala.Tuple3<String, Integer, Integer>> scalaTupleCollection =
new ArrayList<>(20);
- for (int i = 0; i < 20; i++) {
- scalaTupleCollection.add(new
scala.Tuple3<>(UUID.randomUUID().toString(), i, 0));
- }
- try {
- sink.open(new Configuration());
- for (scala.Tuple3<String, Integer, Integer> value :
scalaTupleCollection) {
- sink.invoke(value, SinkContextUtil.forTimestamp(0));
- }
- } finally {
- sink.close();
- }
-
- ResultSet rs =
- cassandraTestEnvironment.executeRequestWithTimeout(
- injectTableName(SELECT_DATA_QUERY));
- List<com.datastax.driver.core.Row> rows = rs.all();
- assertThat(rows).hasSameSizeAs(scalaTupleCollection);
-
- for (com.datastax.driver.core.Row row : rows) {
- scalaTupleCollection.remove(
- new scala.Tuple3<>(
- row.getString(TUPLE_ID_FIELD),
- row.getInt(TUPLE_COUNTER_FIELD),
- row.getInt(TUPLE_BATCHID_FIELD)));
- }
- assertThat(scalaTupleCollection).isEmpty();
- }
-
- @Test
- void testCassandraScalaTuplePartialColumnUpdate() throws Exception {
Review Comment:
This was the only test for `setIgnoreNullFields(true)`, and the tuple and
row sinks still go through that path. Could you port it to a `Tuple3` with
`CassandraTupleSink` instead of dropping it?
##########
flink-connector-cassandra/pom.xml:
##########
@@ -38,16 +38,15 @@ under the License.
<!-- Allow users to pass custom connector versions -->
<properties>
<scala.binary.version>2.12</scala.binary.version>
Review Comment:
Nothing in the module depends on Scala anymore, so this property now only
feeds the `_2.12` suffix on the artifactId. Since 4.0 is a breaking release
anyway, do we want to drop the suffix (here or in its own ticket)?
##########
flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/CassandraConnectorITCase.java:
##########
@@ -289,7 +282,7 @@ protected CassandraTupleWriteAheadSink<Tuple3<String,
Integer, Integer>> createS
return new CassandraTupleWriteAheadSink<>(
injectTableName(INSERT_DATA_QUERY),
TypeExtractor.getForObject(new Tuple3<>("", 0, 0))
- .createSerializer(new ExecutionConfig()),
+ .createSerializer(new SerializerConfigImpl()),
Review Comment:
Heads-up, this predates your change: the five tests inherited from
`WriteAheadSinkTestBase` use JUnit 4 `@Test`, so they never run (CI shows
`Tests run: 11` here). If someone revives them, they'll now fail in `open()`,
since the harness's job config has no checkpoint interval. Worth a follow-up
ticket to move the base to JUnit 5?
##########
flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java:
##########
@@ -223,7 +208,10 @@ public static <IN> CassandraSinkBuilder<IN>
addSink(DataStream<IN> input) {
tupleInput
.getType()
.createSerializer(
-
tupleInput.getExecutionEnvironment().getConfig()));
+ tupleInput
+ .getExecutionEnvironment()
+ .getConfig()
+ .getSerializerConfig()));
Review Comment:
nit: `getExecutionEnvironment().getConfig().getSerializerConfig()` is now
spelled out three times. A local `SerializerConfig` at the top of `addSink`
would shorten all three branches.
##########
flink-connector-cassandra/src/main/java/org/apache/flink/streaming/connectors/cassandra/CassandraSink.java:
##########
@@ -233,23 +221,19 @@ public static <IN> CassandraSinkBuilder<IN>
addSink(DataStream<IN> input) {
rowInput.getType(),
rowInput.getType()
.createSerializer(
-
rowInput.getExecutionEnvironment().getConfig()));
+ rowInput.getExecutionEnvironment()
+ .getConfig()
+ .getSerializerConfig()));
}
if (typeInfo instanceof PojoTypeInfo) {
return new CassandraPojoSinkBuilder<>(
input,
input.getType(),
-
input.getType().createSerializer(input.getExecutionEnvironment().getConfig()));
- }
- if (typeInfo instanceof CaseClassTypeInfo) {
- DataStream<Product> productInput = (DataStream<Product>) input;
- return (CassandraSinkBuilder<IN>)
- new CassandraScalaProductSinkBuilder<>(
- productInput,
- productInput.getType(),
- productInput
- .getType()
-
.createSerializer(input.getExecutionEnvironment().getConfig()));
+ input.getType()
+ .createSerializer(
+ input.getExecutionEnvironment()
+ .getConfig()
+ .getSerializerConfig()));
}
throw new IllegalArgumentException(
"No support for the type of the given DataStream: " +
input.getType());
Review Comment:
nit: Scala case-class streams land here now. Maybe list the supported types
(Tuple, Row, POJO) in the message so the error says what to do?
##########
flink-connector-cassandra/pom.xml:
##########
@@ -341,19 +329,13 @@ under the License.
<version>${byte-buddy.version}</version>
</dependency>
- <!-- For dependency convergence -->
<dependency>
- <groupId>com.esotericsoftware.kryo</groupId>
- <artifactId>kryo</artifactId>
- <version>2.24.0</version>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <version>${byte-buddy.version}</version>
</dependency>
- <!-- For dependency convergence -->
- <dependency>
- <groupId>org.scala-lang</groupId>
- <artifactId>scala-library</artifactId>
- <version>${scala-library.version}</version>
- </dependency>
+
Review Comment:
nit: blank lines left over from the removed dependencies (same at lines 66
and 74).
```suggestion
```
##########
pom.xml:
##########
@@ -42,7 +42,7 @@ under the License.
</scm>
<properties>
- <flink.version>1.20.0</flink.version>
+ <flink.version>2.3.0</flink.version>
Review Comment:
Make sense. But we don't have any support for 2.2 Flink for this connector.
Probably adding this point to release notes also works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]