Hey! I tend to return to StartMiniHS2Cluster from time to time. It's very good for the "change code - compile - run - debug - repeat" way of doing things. >From this point of view, docker image is not an alternative to that. Also, StartMiniHS2Cluster just works, always, moreover, it uses the same minicluster architecture as our qtests (I mean the way it achieves a mini mr/tez/llap/whatever cluster from the hadoop shim). Qtests are also not an alternative: I often need the ability to run queries in random order or in adhoc way without editing a qfile.
I feel that HIVE-27338 <https://issues.apache.org/jira/browse/HIVE-27338> was about to solve licensing problems, and it achieved that, but maybe went too far: I believe we should be able to provide jars to the test classpath, e.g. jars that were downloaded beforehand, even manually. Regards, Laszlo Bodor Stamatis Zampetakis <zabe...@gmail.com> ezt írta (időpont: 2023. aug. 18., P, 13:08): > Hey Zsolt, > > I would divide this discussion into three topics: > > 1. What are the benefits of using the StartMiniHS2Cluster? > 2. What other alternatives are there for testing HS2 with different > metastore DBMS? > 3. How can we make StartMiniHS2Cluster work as before? > > Regarding the point 1, I don't have an answer because I never used > StartMiniHS2Cluster myself. Obviously, other people here are using it so I > would be curious to know in which cases this is useful. > > There are various alternatives for testing HS2 with different metastore > DBMS. > > The first and in my opinion the easiest to use is the classic qtest > framework with the various CLI drivers. Basically, with the work that > Laszlo started in HIVE-21954 it is pretty easy to run any kind of test over > any metastore just by setting the respective system property. > > mvn test -Dtest=TestMiniLlapLocalCliDriver > -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres > > The second is to use our brand new and shiny docker images contributed by > Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like > setup. I haven't played much with this in apache/master but I did use some > similar images in our internal forks and it's pretty easy to get it up and > running. > > The third is the old and classic hive-dev-box [1] started by Zoltan. With > two/three commands you have a Hive cluster like environment and of course > you can choose which DBMS you want for the metastore. > > Regarding point 3, I assume that it is pretty easy to fix by adding the > postgresql (or other JDBC driver) dependency inside the hive-it-unit > module. > > <dependency> > <groupId>org.postgresql</groupId> > <artifactId>postgresql</artifactId> > <optional>true</optional> > </dependency> > > Given that we have other alternatives do we really need to go into this > direction? In fact, do we really need the StartMiniHS2Cluster class? > > Best, > Stamatis > > [1] https://github.com/kgyrtkirk/hive-dev-box > > On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zsolt.miskol...@gmail.com> > wrote: > > > Hey there!Do you know how it is possible to use minihs2 with embedded > > metastore service but postgresql as the metastore database? > > > > Since I'm pretty sure it broke with that change > > https://github.com/apache/hive/pull/4317, handling drivers is changed: > > > > We don't bundle postgresql drivers with the hive build.Firstly, when I > copy > > the postgresql driver jar into the > > > > > packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib > > directory, I can run the schematool as it sees the driver. > > The real issue is with miniHS2Cluster: it is basically a maven test and I > > have really no idea how I can pass the driver to the test. If it would > be a > > simple java command, I would try to pass the -cp argument but I see no > > other way to add extra jar file to the test execution.I want to run > minihs2 > > with this command: mvn test -Dtest=StartMiniHS2Cluster -DminiHS2.run=true > > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP > > -DminiHS2.conf="../../data/conf/llap/hive-site.xml" > > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true > > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it > seems > > the metastore client cannot find the proper driver): > > > > java.lang.RuntimeException: Error applying authorization policy on hive > > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable > to > > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122) > > at > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > > at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243) > > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394) > > at > > > > > org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > > at > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > > at > > > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > > at > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > > at > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > > at > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > > at > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > > at > > > > > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) > > at > > > > > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) > > at > org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) > > at > > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) > > Caused by: java.lang.RuntimeException: > > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > > > > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001) > > at > > > > > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003) > > at > > > > > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135) > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119) > > ... 32 more > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > > > > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033) > > at > > > > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993) > > ... 35 more > > Caused by: java.lang.RuntimeException: Unable to instantiate > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > at > > > > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88) > > at > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96) > > at > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149) > > at > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120) > > at > > > > > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769) > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847) > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827) > > at > > > > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029) > > ... 36 more > > Caused by: java.lang.reflect.InvocationTargetException > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > > > > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) > > ... 43 more > > Caused by: MetaException(message:Got exception: > > org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException: > > No suitable driver) > > at > > > > > org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150) > > at > > > > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309) > > at > > > > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220) > > at > > > > > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120) > > at > > > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154) > > ... 48 more > > > > > > Any help is appreciated, > > Thank you > > >
