david-streamlio opened a new pull request, #93:
URL: https://github.com/apache/pulsar-connectors/pull/93
Fixes #52
### Motivation
The `hdfs3` module previously had only config-validation tests and
mock-based sink tests (`HdfsSequentialSinkTest`, `HdfsTextSinkTest`,
`HdfsStringSinkTest`). Those verify that records are acknowledged against a
Mockito mock, but they point at the local filesystem and never write to or read
back from a real HDFS filesystem — so nothing exercised the sink's actual HDFS
write path end-to-end.
### Modifications
- Added
`hdfs3/src/test/java/org/apache/pulsar/io/hdfs3/sink/HdfsSinkIntegrationTest.java`,
which:
- Starts an in-JVM Hadoop `MiniDFSCluster` in `@BeforeClass` (no Docker /
external services required).
- Generates a `core-site.xml` whose `fs.defaultFS` points at the running
mini cluster and hands it to the sink via `hdfsConfigResources`.
- Opens `HdfsStringSink`, writes 25 records, waits (Awaitility, no
`Thread.sleep` before asserting) for all records to be acked so the sink's
unacked queue is drained, then `close()`s the sink to flush and commit the file.
- Reads the committed file back **from the mini cluster's own
`FileSystem`** and asserts the bytes exactly match what was written.
- Bounded with `@Test(timeOut = 300_000)`.
- `hdfs3/build.gradle.kts`: added
`testImplementation(libs.hadoop.minicluster)` (already present in the version
catalog, pinned to the module's `hadoop3` version 3.5.0). The mini cluster's
embedded HDFS HTTP server requires Jetty 9.x while the shared platform enforces
Jetty 12.x (which removed classes such as `HandlerWrapper`); following the
existing pattern used by the `alluxio` module, the Jetty 12 BOM is excluded
from the enforced platform and the test classpath is pinned to Jetty 9.x via
the existing `jetty9-bom-override` alias. This affects the test classpath only
— the connector is an HDFS client and never loads Jetty at runtime.
### Verifying this change
`./gradlew :hdfs3:test` — full module suite passes (the new test plus the
existing suite), run repeatedly:
```
Gradle suite > Gradle test >
org.apache.pulsar.io.hdfs3.sink.HdfsSinkIntegrationTest >
testRecordsAreWrittenToHdfs PASSED
BUILD SUCCESSFUL
```
Run 3x with `--rerun-tasks -PtestFailFast=false`: all `BUILD SUCCESSFUL`, no
flakes. (An earlier revision was flaky under parallel forks because the sink's
`close()` calls `hsync()` on an already-closed stream when records remain
unacked — a real ordering quirk in
`HdfsAbstractTextFileSink.close()`/`HdfsSyncThread.halt()` that only surfaces
on real HDFS. The test now waits for the queue to drain before closing, making
it deterministic.)
**Mutation check (required):** appending content that was never written to
the expected string makes the test FAIL, proving the assertion genuinely reads
back the real HDFS data rather than passing vacuously:
```
org.apache.pulsar.io.hdfs3.sink.HdfsSinkIntegrationTest >
testRecordsAreWrittenToHdfs FAILED
java.lang.AssertionError: content read back from HDFS must match what
was written to the sink
expected [integration-record-0
integration-record-1
...
```
Reverting the mutation returns the suite to green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
