Hi Team, I agree with Laszlo Bodor, it is a good tool to develop Hive. It worked properly before this PR. My question is how do you provide the missing database jar (e.g. Postgres driver) for StartMiniHS2Cluster, which is run by maven? If it requires to modify the pom file after this change, then there is no difference then just alway reverting this PR mentioned earlier, but it has the risk this will be committed together with the actual work a developer does. I think this is a wrong approach to modify a pom or java file to be able to run it. We must be able to run it without alway modifying.
Regards, Zoltan Ratkai On Mon, Aug 21, 2023 at 10:44 AM László Bodor <bodorlaszlo0...@gmail.com> wrote: > Hey! > > I tend to return to StartMiniHS2Cluster from time to time. It's very good > for the "change code - compile - run - debug - repeat" way of doing things. > From this point of view, docker image is not an alternative to that. Also, > StartMiniHS2Cluster just works, always, moreover, it uses the same > minicluster architecture as our qtests (I mean the way it achieves a mini > mr/tez/llap/whatever cluster from the hadoop shim). Qtests are also not an > alternative: I often need the ability to run queries in random order or in > adhoc way without editing a qfile. > > I feel that HIVE-27338 <https://issues.apache.org/jira/browse/HIVE-27338> > was > about to solve licensing problems, and it achieved that, but maybe went too > far: I believe we should be able to provide jars to the test classpath, > e.g. jars that were downloaded beforehand, even manually. > > Regards, > Laszlo Bodor > > > Stamatis Zampetakis <zabe...@gmail.com> ezt írta (időpont: 2023. aug. 18., > P, 13:08): > > > Hey Zsolt, > > > > I would divide this discussion into three topics: > > > > 1. What are the benefits of using the StartMiniHS2Cluster? > > 2. What other alternatives are there for testing HS2 with different > > metastore DBMS? > > 3. How can we make StartMiniHS2Cluster work as before? > > > > Regarding the point 1, I don't have an answer because I never used > > StartMiniHS2Cluster myself. Obviously, other people here are using it so > I > > would be curious to know in which cases this is useful. > > > > There are various alternatives for testing HS2 with different metastore > > DBMS. > > > > The first and in my opinion the easiest to use is the classic qtest > > framework with the various CLI drivers. Basically, with the work that > > Laszlo started in HIVE-21954 it is pretty easy to run any kind of test > over > > any metastore just by setting the respective system property. > > > > mvn test -Dtest=TestMiniLlapLocalCliDriver > > -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres > > > > The second is to use our brand new and shiny docker images contributed by > > Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like > > setup. I haven't played much with this in apache/master but I did use > some > > similar images in our internal forks and it's pretty easy to get it up > and > > running. > > > > The third is the old and classic hive-dev-box [1] started by Zoltan. With > > two/three commands you have a Hive cluster like environment and of course > > you can choose which DBMS you want for the metastore. > > > > Regarding point 3, I assume that it is pretty easy to fix by adding the > > postgresql (or other JDBC driver) dependency inside the hive-it-unit > > module. > > > > <dependency> > > <groupId>org.postgresql</groupId> > > <artifactId>postgresql</artifactId> > > <optional>true</optional> > > </dependency> > > > > Given that we have other alternatives do we really need to go into this > > direction? In fact, do we really need the StartMiniHS2Cluster class? > > > > Best, > > Stamatis > > > > [1] https://github.com/kgyrtkirk/hive-dev-box > > > > On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zsolt.miskol...@gmail.com > > > > wrote: > > > > > Hey there!Do you know how it is possible to use minihs2 with embedded > > > metastore service but postgresql as the metastore database? > > > > > > Since I'm pretty sure it broke with that change > > > https://github.com/apache/hive/pull/4317, handling drivers is changed: > > > > > > We don't bundle postgresql drivers with the hive build.Firstly, when I > > copy > > > the postgresql driver jar into the > > > > > > > > > packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib > > > directory, I can run the schematool as it sees the driver. > > > The real issue is with miniHS2Cluster: it is basically a maven test > and I > > > have really no idea how I can pass the driver to the test. If it would > > be a > > > simple java command, I would try to pass the -cp argument but I see no > > > other way to add extra jar file to the test execution.I want to run > > minihs2 > > > with this command: mvn test -Dtest=StartMiniHS2Cluster > -DminiHS2.run=true > > > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP > > > -DminiHS2.conf="../../data/conf/llap/hive-site.xml" > > > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true > > > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it > > seems > > > the metastore client cannot find the proper driver): > > > > > > java.lang.RuntimeException: Error applying authorization policy on hive > > > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable > > to > > > instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122) > > > at > > org.apache.hive.service.CompositeService.init(CompositeService.java:59) > > > at > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243) > > > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394) > > > at > > > > > > > > > org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > > > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > > > at > > > > > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > > at > > > > > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > > > at > > > > > > > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > > > at > > > > > > > > > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > > > at > > > > > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > > > at > > > > > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > > > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > > > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > > > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > > > at > > > > > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > > > at > > > > > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > > > at > > > > > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > > > at > > > > > > > > > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > > > at > > > > > > > > > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) > > > at > > > > > > > > > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) > > > at > > org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) > > > at > > > > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) > > > Caused by: java.lang.RuntimeException: > > > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > > at > > > > > > > > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003) > > > at > > > > > > > > > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135) > > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119) > > > ... 32 more > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > > > instantiate > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > > at > > > > > > > > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993) > > > ... 35 more > > > Caused by: java.lang.RuntimeException: Unable to instantiate > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769) > > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847) > > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029) > > > ... 36 more > > > Caused by: java.lang.reflect.InvocationTargetException > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > > at > > > > > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86) > > > ... 43 more > > > Caused by: MetaException(message:Got exception: > > > org.apache.hadoop.hive.metastore.api.MetaException > java.sql.SQLException: > > > No suitable driver) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309) > > > at > > > > > > > > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120) > > > at > > > > > > > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154) > > > ... 48 more > > > > > > > > > Any help is appreciated, > > > Thank you > > > > > >
