[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-618113749 @yanghua @bvaradar @vinothchandar I've fixed this PR, cleaned up a bunch of the code. This test suite now has test cases that one can use to run end to end tests in junit. At the moment, the test suite does not run in docker due to Spark 2.4 bringing in Hive 1.x dependencies and our code using Hive 2.x to spin up local hive server nodes. There is a hacky approach that we are using at Uber which will be upstreamed in the next couple of weeks (https://issues.apache.org/jira/browse/HUDI-830) and as part of that we can discuss how to solve it (the right way to solve without any hacks is to move to spark 3.x since they upgraded the Hive libs there but that might take a while etc). Once that is done, we can even run this as an integration test. The following is my suggestion for a plan for this : 1) Land this PR which provides an initial test suite to test basic end to end functionality. This allows folks to atleast start using this framework to test large PRs but spending minimal amount of time to enhance the test suite (especially since lots of refactoring is happening). Last I checked, @yanghua I think you went over the PR and were okay to merge. @vinothchandar @bvaradar unless we have major concerns on the PR, we can merge it and then take incremental pr's 2) There are many tickets under HUDI-289, all the enhancements to the test suite from Uber will follow in subsequent PR's in the next 2-3 weeks. I'm working with the necessary folks along with some of the enhancements that I want to do in the coming weeks. 3) @yanghua need you to lead the Azure pipelines for the test suite and other tickets assigned to you under the umbrella ticket. Additionally, since the test suite actually tests all end to end functionality, the main class tests take a while to run. Also, without the test suite we are nearing the max time allowed to run unit tests on travis (50 mins), so I've separated out the test suite unit tests into a different job, just like the integration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-585444365 @yanghua merged the pom PR, can you rebase this so we can merge this PR ? I think we have gone through it multiple times, let's have the first version committed so we can come up with more action items around this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-579386272 @yanghua please take a look at this PR once you are back from the new year holidays, we should merge this very soon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-571389593 @yanghua Left a comment in the other PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-568859715 @yanghua thanks for fixing the build. Were you able to give the test-suite a try ? That is : 1) Create your own yaml 2) Run the TestHudiTestSuiteJob 3) Run it in docker vs run it in your own spark cluster if you have one This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-568566995 @yanghua Can you check why the build is failing now ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end
n3nash commented on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-566306717 @yanghua great, thanks. Can you please take a pass at the current PR and leave any comments that we should address ? Also, lets start running the test suite (I'm doing that too) to see if it works as expected in a longer duration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
