jon-wei opened a new pull request #9714: More Hadoop integration tests URL: https://github.com/apache/druid/pull/9714 This PR adds a new set of Hadoop integration tests (not run in Travis CI in this PR). - Added Hadoop counterpart to the native batch wikipedia test, with variations on the partitioning spec. This reads from HDFS and writes to HDFS. - Added tests for Hadoop ingestion + cloud stores (Azure, GCS, S3). - Note that using Azure as deep storage with Hadoop 2.8.5 ingestion does not currently work due to conflicts in the `azure-storage` version that Druid uses (8.6.0) and the one that `hadoop-azure` uses (2.2.0) Some other changes included in this PR: - Enables the management proxy on the main router service, which makes it easier to troubleshoot the integration tests live using the web console - Removes an unneeded `-sha256` parameter from the `openssl genrsa` calls, which was causing problems in my environment with a newer openssl that rejects the unrecognized parameter - Allows setting `extra.datasource.name.suffix` for the integration tests, which is needed for these tests as our cloud store extensions currently don't handle the special characters in the datasource names properly. This PR has: - [x] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [x] added integration tests. - [x] been tested in a test Druid cluster.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
