Re: Docker Hive using tez without hdfs

2024-01-11 Thread Sanjay Gupta 
Thanks Attila & Ayush,
I don't have permission to open Jira ticket yet but I have initiated process.
I have tried with Tez 9.1 and also version 10.2 and same issue.
I have noticed that when I change default hive.execution.engine=mr in
hive-site.xml ( restart hive service ) and after that start  hive cli
and then do set hive.execution.engine=tez on command line and run
query, it doesn't give error.
However when default engine is set to tez in hive-site.xml, hive cli
exits out with error
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> org/apache/tez/dag/api/TezConfiguration

Thanks

On Wed, Jan 10, 2024 at 5:40 AM Attila Turoczy  wrote:
>
> Agree with Ayush.
>
> Back to the original issue, is it not related to the latest Tez fix? As I 
> remember there was an incompatibility issue, which the next tez release will 
> fix. Maybe this is related to that. Sanjay could you please create a JIRA 
> around it for the tracking, and the community or someone from the community 
> will check. (I know most of you don't like the jira but this could help the 
> tracking then a mail thread)
>
> -Attila
>
> On Wed, Jan 10, 2024 at 8:45 AM Ayush Saxena  wrote:
>>
>> Hive on MR3 isn’t an official Apache Hive thing, not even an Apache OS 
>> thing, so, it is a vendor product just being tried to advertised in the 
>> ‘Apache’ Hive space
>>
>> So, it can be all mess, filled with security issues or bugs & we Apache Hive 
>> for the record aren’t responsible for that neither do we endorse usage of 
>> that or anything outside the scope of Apache
>>
>> -Ayush
>>
>> On 10-Jan-2024, at 1:09 PM, Sungwoo Park  wrote:
>>
>> 
>> As far as I know, Hive-Tez supports local mode, but does not standalone mode 
>> (like Spark). Hive-MR3 supports standalone mode, so you can run it in any 
>> type of cluster.
>>
>> --- Sungwoo
>>
>> On Wed, Jan 10, 2024 at 4:22 PM Sanjay Gupta  wrote:
>>>
>>> I can run hive with mr engine in local mode. Does Hive + Tez also
>>> works in standalone mode ?
>>>
>>> On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
>>> >
>>> > Hello,
>>> >
>>> > I don't have an answer to your problem, but if your goal is to quickly 
>>> > test Hive 3 using Docker, there is an alternative way which uses Hive on 
>>> > MR3.
>>> >
>>> > https://mr3docs.datamonad.com/docs/quick/docker/
>>> >
>>> > You can also run Hive on MR3 on Kubernetes.
>>> >
>>> > Thanks,
>>> >
>>> > --- Sungwoo
>>> >
>>> >
>>> >
>>> > On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta  wrote:
>>> >>
>>> >> Hi,
>>> >> Using following docker container to run meta , hiveserver2
>>> >>
>>> >> https://hub.docker.com/r/apache/hive
>>> >> https://github.com/apache/hive/blob/master/packaging/src/docker/
>>> >>
>>> >> I have configured hive-site.xml to se S3
>>> >> When I set in hive.execution.engine to mr hive-site.xml, hive is
>>> >> running fine and I can perform queries but setting to tez fails with
>>> >> error.
>>> >> There is no hdfs but it is running in local mode.
>>> >>
>>> >> 
>>> >> hive.execution.engine
>>> >> tez
>>> >> 
>>> >>
>>> >> Any idea how to fix this issue ?
>>> >>
>>> >> hive
>>> >> SLF4J: Actual binding is of type 
>>> >> [org.apache.logging.slf4j.Log4jLoggerFactory]
>>> >> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>>> >>
>>> >> Logging initialized using configuration in
>>> >> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
>>> >> Async: true
>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >> org/apache/tez/dag/api/TezConfiguration
>>> >> at 
>>> >> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
>>> >> at 
>>> >> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
>>> >> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
>>> >> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> >> at 
>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> >> at 
>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>>> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
>>> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
>>> >> Caused by: java.lang.ClassNotFoundException:
>>> >> org.apache.tez.dag.api.TezConfiguration
>>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> Thanks
>>> >> Sanjay Gupta
>>>
>>>
>>>
>>> --
>>>
>>> Thanks
>>> Sanjay Gupta



-- 

Thanks
Sanjay Gupta

Re: Docker Hive using tez without hdfs

2024-01-10 Thread Attila Turoczy 
Agree with Ayush.

Back to the original issue, is it not related to the latest Tez fix? As I
remember there was an incompatibility issue, which the next tez release
will fix. Maybe this is related to that. Sanjay could you please create a
JIRA around it for the tracking, and the community or someone from the
community will check. (I know most of you don't like the jira but this
could help the tracking then a mail thread)

-Attila

On Wed, Jan 10, 2024 at 8:45 AM Ayush Saxena  wrote:

> Hive on MR3 isn’t an official Apache Hive thing, not even an Apache OS
> thing, so, it is a vendor product just being tried to advertised in the
> ‘Apache’ Hive space
>
> So, it can be all mess, filled with security issues or bugs & we Apache
> Hive for the record aren’t responsible for that neither do we endorse usage
> of that or anything outside the scope of Apache
>
> -Ayush
>
> On 10-Jan-2024, at 1:09 PM, Sungwoo Park  wrote:
>
> 
> As far as I know, Hive-Tez supports local mode, but does not standalone
> mode (like Spark). Hive-MR3 supports standalone mode, so you can run it in
> any type of cluster.
>
> --- Sungwoo
>
> On Wed, Jan 10, 2024 at 4:22 PM Sanjay Gupta  wrote:
>
>> I can run hive with mr engine in local mode. Does Hive + Tez also
>> works in standalone mode ?
>>
>> On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
>> >
>> > Hello,
>> >
>> > I don't have an answer to your problem, but if your goal is to quickly
>> test Hive 3 using Docker, there is an alternative way which uses Hive on
>> MR3.
>> >
>> > https://mr3docs.datamonad.com/docs/quick/docker/
>> >
>> > You can also run Hive on MR3 on Kubernetes.
>> >
>> > Thanks,
>> >
>> > --- Sungwoo
>> >
>> >
>> >
>> > On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta 
>> wrote:
>> >>
>> >> Hi,
>> >> Using following docker container to run meta , hiveserver2
>> >>
>> >> https://hub.docker.com/r/apache/hive
>> >> https://github.com/apache/hive/blob/master/packaging/src/docker/
>> >>
>> >> I have configured hive-site.xml to se S3
>> >> When I set in hive.execution.engine to mr hive-site.xml, hive is
>> >> running fine and I can perform queries but setting to tez fails with
>> >> error.
>> >> There is no hdfs but it is running in local mode.
>> >>
>> >> 
>> >> hive.execution.engine
>> >> tez
>> >> 
>> >>
>> >> Any idea how to fix this issue ?
>> >>
>> >> hive
>> >> SLF4J: Actual binding is of type
>> [org.apache.logging.slf4j.Log4jLoggerFactory]
>> >> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>> >>
>> >> Logging initialized using configuration in
>> >> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
>> >> Async: true
>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >> org/apache/tez/dag/api/TezConfiguration
>> >> at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
>> >> at
>> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
>> >> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
>> >> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
>> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.tez.dag.api.TezConfiguration
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> >>
>> >>
>> >> --
>> >>
>> >> Thanks
>> >> Sanjay Gupta
>>
>>
>>
>> --
>>
>> Thanks
>> Sanjay Gupta
>>
>

Re: Docker Hive using tez without hdfs

2024-01-10 Thread Zoltán Rátkai 
Hi,

I am not sure if the official built docker contains TEZ, but if you try to
build it by yourself, you can have a look at here:
https://github.com/apache/hive/blob/master/packaging/src/docker/Dockerfile#L41

To use TEZ you need to place it inside the docker container and configure
it:
Download from here:
https://archive.apache.org/dist/tez/0.10.2/apache-tez-0.10.2-bin.tar.gz

and configure it with:
export TEZ_HOME="/tez"
export TEZ_CONF_DIR="/hive/conf/"
export HADOOP_CLASSPATH="$TEZ_HOME/*:$TEZ_HOME/lib/*:$HADOOP_CLASSPATH"

The above Dockerfile seems to do all that job, so try to build docker image
with with the official Dockerfile, it should work.

Regards,

Zoltan Ratkai

On Wed, Jan 10, 2024 at 8:45 AM Ayush Saxena  wrote:

> Hive on MR3 isn’t an official Apache Hive thing, not even an Apache OS
> thing, so, it is a vendor product just being tried to advertised in the
> ‘Apache’ Hive space
>
> So, it can be all mess, filled with security issues or bugs & we Apache
> Hive for the record aren’t responsible for that neither do we endorse usage
> of that or anything outside the scope of Apache
>
> -Ayush
>
> On 10-Jan-2024, at 1:09 PM, Sungwoo Park  wrote:
>
> 
> As far as I know, Hive-Tez supports local mode, but does not standalone
> mode (like Spark). Hive-MR3 supports standalone mode, so you can run it in
> any type of cluster.
>
> --- Sungwoo
>
> On Wed, Jan 10, 2024 at 4:22 PM Sanjay Gupta  wrote:
>
>> I can run hive with mr engine in local mode. Does Hive + Tez also
>> works in standalone mode ?
>>
>> On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
>> >
>> > Hello,
>> >
>> > I don't have an answer to your problem, but if your goal is to quickly
>> test Hive 3 using Docker, there is an alternative way which uses Hive on
>> MR3.
>> >
>> > https://mr3docs.datamonad.com/docs/quick/docker/
>> >
>> > You can also run Hive on MR3 on Kubernetes.
>> >
>> > Thanks,
>> >
>> > --- Sungwoo
>> >
>> >
>> >
>> > On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta 
>> wrote:
>> >>
>> >> Hi,
>> >> Using following docker container to run meta , hiveserver2
>> >>
>> >> https://hub.docker.com/r/apache/hive
>> >> https://github.com/apache/hive/blob/master/packaging/src/docker/
>> >>
>> >> I have configured hive-site.xml to se S3
>> >> When I set in hive.execution.engine to mr hive-site.xml, hive is
>> >> running fine and I can perform queries but setting to tez fails with
>> >> error.
>> >> There is no hdfs but it is running in local mode.
>> >>
>> >> 
>> >> hive.execution.engine
>> >> tez
>> >> 
>> >>
>> >> Any idea how to fix this issue ?
>> >>
>> >> hive
>> >> SLF4J: Actual binding is of type
>> [org.apache.logging.slf4j.Log4jLoggerFactory]
>> >> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>> >>
>> >> Logging initialized using configuration in
>> >> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
>> >> Async: true
>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >> org/apache/tez/dag/api/TezConfiguration
>> >> at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
>> >> at
>> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
>> >> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
>> >> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
>> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.tez.dag.api.TezConfiguration
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> >>
>> >>
>> >> --
>> >>
>> >> Thanks
>> >> Sanjay Gupta
>>
>>
>>
>> --
>>
>> Thanks
>> Sanjay Gupta
>>
>

Re: Docker Hive using tez without hdfs

2024-01-09 Thread Ayush Saxena 
Hive on MR3 isn’t an official Apache Hive thing, not even an Apache OS thing, so, it is a vendor product just being tried to advertised in the ‘Apache’ Hive spaceSo, it can be all mess, filled with security issues or bugs & we Apache Hive for the record aren’t responsible for that neither do we endorse usage of that or anything outside the scope of Apache-AyushOn 10-Jan-2024, at 1:09 PM, Sungwoo Park  wrote:As far as I know, Hive-Tez supports local mode, but does not standalone mode (like Spark). Hive-MR3 supports standalone mode, so you can run it in any type of cluster.--- SungwooOn Wed, Jan 10, 2024 at 4:22 PM Sanjay Gupta  wrote:I can run hive with mr engine in local mode. Does Hive + Tez also
works in standalone mode ?

On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
>
> Hello,
>
> I don't have an answer to your problem, but if your goal is to quickly test Hive 3 using Docker, there is an alternative way which uses Hive on MR3.
>
> https://mr3docs.datamonad.com/docs/quick/docker/
>
> You can also run Hive on MR3 on Kubernetes.
>
> Thanks,
>
> --- Sungwoo
>
>
>
> On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta  wrote:
>>
>> Hi,
>> Using following docker container to run meta , hiveserver2
>>
>> https://hub.docker.com/r/apache/hive
>> https://github.com/apache/hive/blob/master/packaging/src/docker/
>>
>> I have configured hive-site.xml to se S3
>> When I set in hive.execution.engine to mr hive-site.xml, hive is
>> running fine and I can perform queries but setting to tez fails with
>> error.
>> There is no hdfs but it is running in local mode.
>>
>>     
>>         hive.execution.engine
>>         tez
>>     
>>
>> Any idea how to fix this issue ?
>>
>> hive
>> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
>> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>>
>> Logging initialized using configuration in
>> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
>> Async: true
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/tez/dag/api/TezConfiguration
>> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
>> at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.tez.dag.api.TezConfiguration
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>
>>
>> --
>>
>> Thanks
>> Sanjay Gupta



-- 

Thanks
Sanjay Gupta

Re: Docker Hive using tez without hdfs

2024-01-09 Thread Sungwoo Park 
As far as I know, Hive-Tez supports local mode, but does not standalone
mode (like Spark). Hive-MR3 supports standalone mode, so you can run it in
any type of cluster.

--- Sungwoo

On Wed, Jan 10, 2024 at 4:22 PM Sanjay Gupta  wrote:

> I can run hive with mr engine in local mode. Does Hive + Tez also
> works in standalone mode ?
>
> On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
> >
> > Hello,
> >
> > I don't have an answer to your problem, but if your goal is to quickly
> test Hive 3 using Docker, there is an alternative way which uses Hive on
> MR3.
> >
> > https://mr3docs.datamonad.com/docs/quick/docker/
> >
> > You can also run Hive on MR3 on Kubernetes.
> >
> > Thanks,
> >
> > --- Sungwoo
> >
> >
> >
> > On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta  wrote:
> >>
> >> Hi,
> >> Using following docker container to run meta , hiveserver2
> >>
> >> https://hub.docker.com/r/apache/hive
> >> https://github.com/apache/hive/blob/master/packaging/src/docker/
> >>
> >> I have configured hive-site.xml to se S3
> >> When I set in hive.execution.engine to mr hive-site.xml, hive is
> >> running fine and I can perform queries but setting to tez fails with
> >> error.
> >> There is no hdfs but it is running in local mode.
> >>
> >> 
> >> hive.execution.engine
> >> tez
> >> 
> >>
> >> Any idea how to fix this issue ?
> >>
> >> hive
> >> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> >> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
> >>
> >> Logging initialized using configuration in
> >> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
> >> Async: true
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> org/apache/tez/dag/api/TezConfiguration
> >> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
> >> at
> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
> >> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
> >> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.tez.dag.api.TezConfiguration
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> >>
> >>
> >> --
> >>
> >> Thanks
> >> Sanjay Gupta
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>

Re: Docker Hive using tez without hdfs

2024-01-09 Thread Sanjay Gupta 
I can run hive with mr engine in local mode. Does Hive + Tez also
works in standalone mode ?

On Tue, Jan 9, 2024 at 11:08 PM Sungwoo Park  wrote:
>
> Hello,
>
> I don't have an answer to your problem, but if your goal is to quickly test 
> Hive 3 using Docker, there is an alternative way which uses Hive on MR3.
>
> https://mr3docs.datamonad.com/docs/quick/docker/
>
> You can also run Hive on MR3 on Kubernetes.
>
> Thanks,
>
> --- Sungwoo
>
>
>
> On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta  wrote:
>>
>> Hi,
>> Using following docker container to run meta , hiveserver2
>>
>> https://hub.docker.com/r/apache/hive
>> https://github.com/apache/hive/blob/master/packaging/src/docker/
>>
>> I have configured hive-site.xml to se S3
>> When I set in hive.execution.engine to mr hive-site.xml, hive is
>> running fine and I can perform queries but setting to tez fails with
>> error.
>> There is no hdfs but it is running in local mode.
>>
>> 
>> hive.execution.engine
>> tez
>> 
>>
>> Any idea how to fix this issue ?
>>
>> hive
>> SLF4J: Actual binding is of type 
>> [org.apache.logging.slf4j.Log4jLoggerFactory]
>> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>>
>> Logging initialized using configuration in
>> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
>> Async: true
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/tez/dag/api/TezConfiguration
>> at 
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
>> at 
>> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.tez.dag.api.TezConfiguration
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>
>>
>> --
>>
>> Thanks
>> Sanjay Gupta



-- 

Thanks
Sanjay Gupta

Re: Docker Hive using tez without hdfs

2024-01-09 Thread Sungwoo Park 
Hello,

I don't have an answer to your problem, but if your goal is to quickly test
Hive 3 using Docker, there is an alternative way which uses Hive on MR3.

https://mr3docs.datamonad.com/docs/quick/docker/

You can also run Hive on MR3 on Kubernetes.

Thanks,

--- Sungwoo



On Wed, Jan 10, 2024 at 3:25 PM Sanjay Gupta  wrote:

> Hi,
> Using following docker container to run meta , hiveserver2
>
> https://hub.docker.com/r/apache/hive
> https://github.com/apache/hive/blob/master/packaging/src/docker/
>
> I have configured hive-site.xml to se S3
> When I set in hive.execution.engine to mr hive-site.xml, hive is
> running fine and I can perform queries but setting to tez fails with
> error.
> There is no hdfs but it is running in local mode.
>
> 
> hive.execution.engine
> tez
> 
>
> Any idea how to fix this issue ?
>
> hive
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71
>
> Logging initialized using configuration in
> jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
> Async: true
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/tez/dag/api/TezConfiguration
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
> at
> org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.tez.dag.api.TezConfiguration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>
>
> --
>
> Thanks
> Sanjay Gupta
>

Docker Hive using tez without hdfs

2024-01-09 Thread Sanjay Gupta 
Hi,
Using following docker container to run meta , hiveserver2

https://hub.docker.com/r/apache/hive
https://github.com/apache/hive/blob/master/packaging/src/docker/

I have configured hive-site.xml to se S3
When I set in hive.execution.engine to mr hive-site.xml, hive is
running fine and I can perform queries but setting to tez fails with
error.
There is no hdfs but it is running in local mode.


hive.execution.engine
tez


Any idea how to fix this issue ?

hive
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 03368207-1904-4c4c-b63e-b29dd28e0a71

Logging initialized using configuration in
jar:file:/opt/hive/lib/hive-common-3.1.3.jar!/hive-log4j2.properties
Async: true
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/tez/dag/api/TezConfiguration
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
at 
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
Caused by: java.lang.ClassNotFoundException:
org.apache.tez.dag.api.TezConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)


-- 

Thanks
Sanjay Gupta