This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 3436381 [FLINK-20659][yarn][tests] Add flink-yarn-test README
3436381 is described below
commit 3436381117ef8fb21f7e02ef63d6d6ec07792203
Author: Matthias Pohl <[email protected]>
AuthorDate: Wed Feb 24 18:15:32 2021 +0100
[FLINK-20659][yarn][tests] Add flink-yarn-test README
---
flink-yarn-tests/README.md | 17 +++++++++++++++++
.../flink/yarn/YARNSessionCapacitySchedulerITCase.java | 14 ++++++++++++++
2 files changed, 31 insertions(+)
diff --git a/flink-yarn-tests/README.md b/flink-yarn-tests/README.md
new file mode 100644
index 0000000..998b03d
--- /dev/null
+++ b/flink-yarn-tests/README.md
@@ -0,0 +1,17 @@
+# Flink YARN tests
+
+`flink-yarn-test` collects test cases which are deployed to a local Apache
Hadoop YARN cluster.
+There are several things to consider when running these tests locally:
+
+* `YarnTestBase` spins up a `MiniYARNCluster`. This cluster spawns processes
outside of the IDE's JVM
+ to run the workers on. `JAVA_HOME` needs to be set to make this work.
+* The Flink cluster within each test is deployed using the `flink-dist`
binaries. Any changes made
+ to the code will only take effect after rebuilding the `flink-dist` module.
+* Each `YARN*ITCase` will have a local working directory for resources like
logs to be stored. These
+ working directories are located in `flink-yarn-tests/target/` (see
+ `find flink-yarn-tests/target -name "*.err" -or -name "*.out"` for the
test's output).
+* There is a known problem causing test instabilities due to our usage of
Hadoop 2.8.3 executing the
+ tests. This is caused by a bug
[YARN-7007](https://issues.apache.org/jira/browse/YARN-7007) that
+ got fixed in [Hadoop
2.8.6](https://issues.apache.org/jira/projects/YARN/versions/12344056). See
+ [FLINK-15534](https://issues.apache.org/jira/browse/FLINK-15534) for further
details on the
+ related discussion.
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java
index 1f7d226..b8b328a 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java
@@ -232,6 +232,10 @@ public class YARNSessionCapacitySchedulerITCase extends
YarnTestBase {
* <p>This ensures that with (any) pre-allocated off-heap memory by us,
there is some off-heap
* memory remaining for Flink's libraries. Creating task managers will
thus fail if no off-heap
* memory remains.
+ *
+ * @throws NullPointerException There is a known Hadoop bug (YARN-7007)
that got fixed in Hadoop
+ * 2.8.6 but might cause test instabilities. See
FLINK-20659/FLINK-15534 for further
+ * information.
*/
@Test
public void perJobYarnClusterOffHeap() throws Exception {
@@ -289,6 +293,10 @@ public class YARNSessionCapacitySchedulerITCase extends
YarnTestBase {
*
* <p><b>Hint: </b> If you think it is a good idea to add more assertions
to this test, think
* again!
+ *
+ * @throws NullPointerException There is a known Hadoop bug (YARN-7007)
that got fixed in Hadoop
+ * 2.8.6 but might cause test instabilities. See
FLINK-13009/FLINK-15534 for further
+ * information.
*/
@Test
public void
@@ -451,6 +459,9 @@ public class YARNSessionCapacitySchedulerITCase extends
YarnTestBase {
* Test deployment to non-existing queue & ensure that the system logs a
WARN message for the
* user. (Users had unexpected behavior of Flink on YARN because they
mistyped the target queue.
* With an error message, we can help users identifying the issue)
+ *
+ * @throws NullPointerException There is a known Hadoop bug (YARN-7007)
that got fixed in Hadoop
+ * 2.8.6 but might cause test instabilities. See FLINK-15534 for
further information.
*/
@Test
public void testNonexistingQueueWARNmessage() throws Exception {
@@ -493,6 +504,9 @@ public class YARNSessionCapacitySchedulerITCase extends
YarnTestBase {
/**
* Test per-job yarn cluster with the parallelism set at the CliFrontend
instead of the YARN
* client.
+ *
+ * @throws NullPointerException There is a known Hadoop bug (YARN-7007)
that got fixed in Hadoop
+ * 2.8.6 but might cause test instabilities. See FLINK-15534 for
further information.
*/
@Test
public void perJobYarnClusterWithParallelism() throws Exception {
