Hi All, I have been building the tag gobblin_0.9.0 with kafka 9, here are the changes that I did
1) vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin$ git diff diff --git a/gobblin-distribution/gobblin-flavor-custom.gradle b/gobblin-distribution/gobblin-flavor-custom.gradle index 5b363bd45..a890c4944 100644 --- a/gobblin-distribution/gobblin-flavor-custom.gradle +++ b/gobblin-distribution/gobblin-flavor-custom.gradle @@ -1,6 +1,6 @@ dependencies { // Example jobs // compile project(':gobblin-example') - + compile project(':gobblin-modules:gobblin-kafka-09') } diff --git a/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep b/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep deleted file mode 100644 index e69de29bb..000000000 diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep b/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep deleted file mode 100644 index e69de29bb..000000000 diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep b/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep deleted file mode 100644 index e69de29bb..000000000 diff --git a/settings.gradle b/settings.gradle index c63463e0f..ac8db6940 100644 --- a/settings.gradle +++ b/settings.gradle @@ -41,7 +41,7 @@ def modules = ['gobblin-admin', // Disable jacoco for now as Kafka 0.8 is the default version and jacoco does not like the same classes // being declared in different modules def jacocoBlacklist = new HashSet([ - "gobblin-modules:gobblin-kafka-09" + "gobblin-modules:gobblin-kafka-08" ]) modules.each { module -> 2) I also added the gobblin-flavor-custom.gradle in the root directory which is vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin Finally I called ./gradlew -PgobblinFlavor=custom assemble It created the gobblin distribution with the Kakfka0.9. When I run the build which includes the test I got these issues 1) gobblin.metastore.DatabaseJobHistoryStoreV100Test:setUp failed with following error de.flapdoodle.embed.process.exceptions.DistributionException: java.io.IOException: File /home/vicky/.embedmysql/MySQL-5.6/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:69) at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:49) at com.wix.mysql.EmbeddedMysql.<init>(EmbeddedMysql.java:39) at com.wix.mysql.EmbeddedMysql$Builder.start(EmbeddedMysql.java:131) at gobblin.metastore.testing.TestMetastoreDatabaseServer.<init>(TestMetastoreDatabaseServer.java:94) at gobblin.metastore.testing.TestMetastoreDatabaseFactory.ensureDatabaseExists(TestMetastoreDatabaseFactory.java:72) at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(TestMetastoreDatabaseFactory.java:52) at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(TestMetastoreDatabaseFactory.java:47) at gobblin.metastore.DatabaseJobHistoryStoreTest.setUp(DatabaseJobHistoryStoreTest.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) I was able to resolve this by downloading the mysql from here https://downloads.mysql.com/archives/get/file/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz And copying it at the location /home/vicky/.embedmysql/MySQL-5.6/. I did not investigate if we could fix it by some configuration, I just fixed it in this way. 2) Next failure is coming as ******************************************************************************************************** :gobblin-yarn:test :gobblin-modules:gobblin-kafka-08:compileTestJava FAILED > Building 75% > :gobblin-runtime:test > 58 tests completed FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':gobblin-modules:gobblin-kafka-08:compileTestJava'. > Compilation failed; see the compiler error output for details. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 5 mins 19.311 secs ******************************************************************************************************** Since I have blacklisted the kafka-08 in the setting.gradle so it might be giving the compilation error. Can someone try to run the build with the kafak-09 flavor on gobblin0.9 and let me know? I am expecting the build/run to work smoothly however it is not working as expected. Thanks, Vicky