Yes exactly.
This is what is recommended, cause hive on Spark has little interest.
However there is nothing enforcing not todo it.
Important to me cause i sit here and work on grassroots marrieng hive on
kafka-streams.
Owen O'Malley schrieb am Mi., 24. Aug. 2022, 18:51:
> Hive on Sp
Hive on Spark is not recommended. The recommended path is to use either Tez
or LLAP. If you already are using Spark 3, it would be far easier to use
Spark SQL.
.. Owen
On Wed, Aug 24, 2022 at 3:46 AM Fred Bai wrote:
> Hi everyone:
>
> Do we have any support for Hive on Spark? I
tters can do what you are trying to do.
> Use canned solutions such as confluence or AWS EMR and save yourself weeks of
> wasted effort.
>
> Hernán
> On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai
> wrote:
>
>
> Hi everyone:
>
> Do we have any support
tters can do what you are trying to do.
> Use canned solutions such as confluence or AWS EMR and save yourself weeks of
> wasted effort.
>
> Hernán
> On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai
> wrote:
>
>
> Hi everyone:
>
> Do we have any support for Hive o
, 2022 at 08:46:30 PM PDT, Fred Bai
wrote:
Hi everyone:
Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark
version is 3.X.
I found Hive incompatible with Spark3, I modify a lot of code to be compatible.
Hive on Spark has deprecated?
And. Hive on Spark is very slow
Hi everyone:
Do we have any support for Hive on Spark? I need Hive on Spark, but my
Spark version is 3.X.
I found Hive incompatible with Spark3, I modify a lot of code to be
compatible.
Hive on Spark has deprecated?
And. Hive on Spark is very slow when the job executes.
Hive On Spark has never been a one on one replacement for Hive on MR. Hive
on Spark has/had several issues and could not fully cover all use-cases.
Therefore Hive On Spark has pretty much been deprecated in favour of Hive
on Tez. It offers the scalability advantages of Hive on MR with much
we have been running hive3 with tez engine.
On Tue, Jun 21, 2022 at 9:19 AM second_co...@yahoo.com <
second_co...@yahoo.com> wrote:
> Hello team,
>
> The default Hive is using Hadoop map reduce. May i know anyone
> successfully swap the engine by running spark operator/cluster? Any guide
> or
-v
> properties|grep -v q.out`
>
>
> On 1/28/22 2:32 PM, Stamatis Zampetakis wrote:
> > Hi team,
> >
> > Almost one year has passed since the last exchange in this discussion and
> > if I am not wrong there has been no effort to revive Hive-on-Spark. To be
> &
par with
Spark-sql?.
We can't use spark-sql output because it converts CHAR or VARCHAR data
types to STRING in the output which is not desirable in our use case. This
behavior is fixed in Spark 3.x but we can't upgrade the Spark version at
the moment.
Versions used:
Hive: 2.1.1
Spark: 2.4.3
I have a n
passed since the last exchange in this discussion and
if I am not wrong there has been no effort to revive Hive-on-Spark. To be
more precise, I don't think I have seen any Spark related JIRA for quite
some time now and although I don't want to rush into conclusions, there
does not seem to be any
Hi team,
Almost one year has passed since the last exchange in this discussion and
if I am not wrong there has been no effort to revive Hive-on-Spark. To be
more precise, I don't think I have seen any Spark related JIRA for quite
some time now and although I don't want to rush into conclusions
Well I have seen this type of error before.
I tend to create the table in hive first and alter it in spark if needed.
This is spark 3.1.1 with Hive (version 3.1.1)
0: jdbc:hive2://rhes75:10099/default> create table my_table2 (col1 int,
col2 int)
0: jdbc:hive2://rhes75:10099/default> de
Hi there.
I also posted this problem in the spark list. I am no sure this is a
spark or a hive metastore problem. Or if there is some metastore tunning
configuration as workaround.
Spark can't see hive schema updates partly because it stores the schema
in a weird way in hive metastore.
1
hive on spark + sentry
jdbc:hive2://hiveser:1/;user=ajxtj;password=123456;hive.server2.proxy.user=jztwk
pro.put("hiveconf:spark.yarn.queue","root.jzyc");
I use yarn pool root.jzyc
but root.jzc only hive and ajxtj can use.
so I want to use jztkw to submit root.jzyc
Thank you so much Mich! Although a bit older, this is the most detailed
comparison I’ve read on the subject. Thanks again.
Regards,
-Manu
From: Mich Talebzadeh
Sent: Tuesday, October 06, 2020 12:37 PM
To: user
Subject: Re: Hive using Spark engine vs native spark with hive integration
Hi Manu,
In the past (July 2016), I made a presentation organised by then
Hortonworks in London titled "Query Engines for Hive: MR, Spark, Tez with
LLAP – Considerations! "
The PDF presentation is here
<https://talebzadehmich.files.wordpress.com/2016/08/hive_on_spark_only.pdf>
hi,
if you are already running hive with tez,the perf gain won't be obvious
camparing with spark.
I'd recommend experimenting with spark on something new until a better
understanding is formed
Manu Jacob 于2020年10月6日 周二23:47写道：
> Hi All,
>
>
>
> Not sure if I need to ask this q
Hi All,
Not sure if I need to ask this question on hive community or spark community.
We have a set of hive scripts that runs on EMR (Tez engine). We would like to
experiment by moving some of it onto Spark. We are planning to experiment with
two options.
1. Use the current code based
; Hello Hive Users.
>>
>> I am interested in gathering some feedback on the adoption of
>> Hive-on-Spark.
>>
>> Does anyone care to volunteer their usage information and would you be
>> open to removing it in favor of Hive-on-Tez in subsequent releases of Hive?
Why would it be this way instead of the other way around?
On Mon, 27 Jul 2020 at 12:27, David wrote:
> Hello Hive Users.
>
> I am interested in gathering some feedback on the adoption of
> Hive-on-Spark.
>
> Does anyone care to volunteer their usage information and w
Hello Hive Users.
I am interested in gathering some feedback on the adoption of Hive-on-Spark.
Does anyone care to volunteer their usage information and would you be open
to removing it in favor of Hive-on-Tez in subsequent releases of Hive?
If you are on MapReduce still, would you be open
Hello,
May I ask about the Hive on Spark 3.x upgrade plan?
I found the newest dependent Spark version is 2.4.5 on Master Branch.
Thanks,
--- wjl
"Hive on Spark" uses Spark purely as execution engine. It would not get the
benefits of codegen and other optimizations of Spark.
If it is mainly for testing, OOTB parameters should work without issues.
However, Tez has lot better edge than Hive on Spark.
Some of the areas where Hiv
Hi Rajesh,
I'm trying to further my understanding of the various interactions and
set-ups for Hive + Spark
My understanding so far is that running queries against the
SparkThriftServer uses the SparkSQL engine whereas the HiveServer2 + Hive +
Spark execution engine uses Hive primitives and only
ive using Spark as the execution engine and I'm
> pretty confused with the set up.
>
> For reference I'm using AWS EMR.
>
> First, I'm confused at the difference between running Hive with Spark as
> its execution engine sending queries to Hive using HiveServer2 (Thrift),
> and
Hi there,
I would like to run Hive using Spark as the execution engine and I'm pretty
confused with the set up.
For reference I'm using AWS EMR.
First, I'm confused at the difference between running Hive with Spark as
its execution engine sending queries to Hive using HiveServer2 (Thrift
Yes I build the same way as you suggested but no luck.
Regards,
Sachin Janani
On Tue, Jun 19, 2018 at 7:13 PM, Sahil Takiar
wrote:
> You should be building Spark without Hive. For Spark 2.3.0, the command is:
>
> ./dev/make-distribution.sh --name "hadoop2-without-hive" --t
You should be building Spark without Hive. For Spark 2.3.0, the command is:
./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided,orc-provided
<https://cwiki.apache.org/confluence/display/Hive/hadoop-2.7,parquet-pro
As per this link for
>> spark-2.3.0 we need to use hive master instead of hive 3.0.0. Also we
>> need to custom build spark without hive dependencies but after trying
>> all this it shows some compatibility issues.
>>
>>
>> Regards,
>> Sachin Janani
>&
> need to custom build spark without hive dependencies but after trying
> all this it shows some compatibility issues.
>
>
> Regards,
> Sachin Janani
>
> On Tue, Jun 19, 2018 at 5:02 PM, Sahil Takiar
> wrote:
> > Yes, Hive 3.0.0 works with Spark 2.3.0 - this section of
This is the same link which I followed. As per this link for
spark-2.3.0 we need to use hive master instead of hive 3.0.0. Also we
need to custom build spark without hive dependencies but after trying
all this it shows some compatibility issues.
Regards,
Sachin Janani
On Tue, Jun 19, 2018 at 5
Yes, Hive 3.0.0 works with Spark 2.3.0 - this
<https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-VersionCompatibility>
section of the wiki has details on which Hive releases support which Spark
versions.
On Tue, Jun 19, 2018 at 5
Hi,
I am trying to run hive on spark by following the steps mentioned
here-
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
, but getting many compatibility issues like NoSuchMethodError,
NoSuchFieldException etc. So just need to know if it works and whether
gt; wrote:
>
> so... i made some progress after much copying of jar files around (as
> alluded to by Gopal previously on this thread).
>
>
> following the instructions here:
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
>
> and doi
so... i made some progress after much copying of jar files around (as
alluded to by Gopal previously on this thread).
following the instructions here:
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
and doing this as instructed will leave off about a dozen
lity issue.
i know. i know. no surprise here.
so i guess i just got to the point where everybody else is... build spark
w/o hive.
lemme see what happens next.
On Wed, Sep 27, 2017 at 7:41 PM, Stephen Sprague <sprag...@gmail.com> wrote:
> thanks. I haven't had a chance to dig into thi
tSparkMemoryAndCores(SetSparkReducerParallelism.java:236)
>> [hive-exec-2.3.0.jar:2.3.0]
>>
>>
>> i'll dig some more tomorrow.
>>
>> On Tue, Sep 26, 2017 at 8:23 PM, Stephen Sprague <sprag...@gmail.com>
>> wrote:
>>
>>> oh. i missed Gopa
..@gmail.com>
> wrote:
>
>> oh. i missed Gopal's reply. oy... that sounds foreboding. I'll keep you
>> posted on my progress.
>>
>> On Tue, Sep 26, 2017 at 4:40 PM, Gopal Vijayaraghavan <gop...@apache.org>
>> wrote:
>>
>>> Hi,
>>
ogress.
>
> On Tue, Sep 26, 2017 at 4:40 PM, Gopal Vijayaraghavan <gop...@apache.org>
> wrote:
>
>> Hi,
>>
>> > org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a
>> spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed
>&g
a
> spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed
> to create spark client.
>
> I get inexplicable errors with Hive-on-Spark unless I do a three step
> build.
>
> Build Hive first, use that version to build Spark, use that Spark version
> to rebuild Hive.
>
>
odAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Met
Hi,
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to get a spark
> session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> spark client.
I get inexplicable errors with Hive-on-Spark unless I do a three step build.
Build Hive first, use tha
essorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
>
> I bugs me that that class is
I bugs me that that class is in spark-core_2.11-2.2.0.jar yet so seemingly
out of reach. :(
On Tue, Sep 26, 2017 at 2:44 PM, Sahil Takiar <takiar.sa...@gmail.com>
wrote:
> Hey Stephen,
>
> Can you send the full stack trace for the NoClassDefFoundError? For Hive
> 2.3.0, we only suppor
Hey Stephen,
Can you send the full stack trace for the NoClassDefFoundError? For Hive
2.3.0, we only support Spark 2.0.0. Hive may work with more recent versions
of Spark, but we only test with Spark 2.0.0.
--Sahil
On Tue, Sep 26, 2017 at 2:35 PM, Stephen Sprague <sprag...@gmail.com>
* i've installed hive 2.3 and spark 2.2
* i've read this doc plenty of times ->
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
* i run this query:
hive --hiveconf hive.root.logger=DEBUG,console -e 'set
hive.execution.engine=spark; select date_key, co
Xuefu is planning to give a talk on Hive-on-Spark @Uber the user meetup
this week. We can check if can share the presentation on this list for
folks who can't attend the meetup.
https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/
On Mon, Aug 21, 2017 at 11:44 PM, peter zhang
Hi All,
Has anybody used hive on spark in your production environment? How
does it's the stability and performance compared with spark sql?
Hope anybody can share your experience.
Thanks in advance!
I meet the same problem,it seems JavaSparkListener has been delete in spark 2.
But I see someone using hive 1.2.1 with spark 2 is ok. I haven't try yet.
-- --
??: "Stephen Sprague"<sprag...@gmail.com>;
: 2017??3??18??(
ne thing that I don't appreciate from those who promote and DOCUMENT
> spark on hive is that, seemingly, there is absolutely no evidence seen that
> says that hive on spark WORKS.
> As a matter of fact, after a lot of pain, I noticed it is not supported by
> just about anybody.
>
> If
Stephan,
Thanks for the response.
The one thing that I don't appreciate from those who promote and DOCUMENT spark
on hive is that, seemingly, there is absolutely no evidence seen that says that
hive on spark WORKS. As a matter of fact, after a lot of pain, I noticed it is
not supported by just
he tez
>> instead of a spark engine for my particular case.
>>
>> But the journey is what I want to share with you.
>> The big data apache tools and libraries such as Hive, Tez, Spark, Hadoop
>> , Parquet etc etc are not interchangeable as we would like to think. There
>>
che tez instead
> of a spark engine for my particular case.
>
> But the journey is what I want to share with you.
> The big data apache tools and libraries such as Hive, Tez, Spark, Hadoop ,
> Parquet etc etc are not interchangeable as we would like to think. There
> are very limit
with you.The big data apache tools and
libraries such as Hive, Tez, Spark, Hadoop , Parquet etc etc are not
interchangeable as we would like to think. There are very limited combinations
for very specific versions. This is why tools like Ambari can be useful. Ambari
sets a path of combos of versions
; hi guys,
> wondering where we stand with Hive On Spark these days?
>
> i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental
> versions) and running up against this class not found:
>
> java.lang.NoClassDefFoundError: org/apache/spark/JavaSparkListener
>
>
> sear
hi guys,
wondering where we stand with Hive On Spark these days?
i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental
versions) and running up against this class not found:
java.lang.NoClassDefFoundError: org/apache/spark/JavaSparkListener
searching the Cyber i find this:
1
Naresh,
The properties hive.spark.job.monitor.timeout and hive.spark.client.server.
connect.timeout in hive-site.xml control Hive on Spark timeouts. Details at
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Spark
Ian Cook
Cloudera
On Thu, Feb
Hello,
i am facing this issue "Job hasn't been submitted after 61s. Aborting it."
when i am running multiple hive queries.
Details: (Hive on Spark)
I am using spark dynamic allocation and external shuffle service (yarn)
Assume one queries is using all of the resources in the cluste
il" cannot go
mapjoin..
and I DEBUG the code,:
in hive-on-spark:
(1)(select a.src_ip,b.appid from small_tbl a join im b on a.src_ip=b.src_ip)
->>> MapWork.getMapredLocalWork() is OK,there is one
MapRedLocalWork Object
(2) the result of the previous stage na
Done. Welcome to the Hive wiki team, Kelly, and happy new year!
-- Lefty
On Mon, Jan 2, 2017 at 5:40 PM, Zhang, Liyun <liyun.zh...@intel.com> wrote:
> Hi
>
> I want to update wiki<https://cwiki.apache.org/
> confluence/display/Hive/Hive+on+Spark%3A+Getting+Starte
Hi
I want to update
wiki<https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started>
of hive on spark because HIVE-8373, my
Confluence<https://cwiki.apache.org/confluence/signup.action> username is
kellyzly, please provide the privilege to me to update
Hi
I want to update
wiki<https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started>
of hive on spark because HIVE-8373, my
Confluence<https://cwiki.apache.org/confluence/signup.action> username is
kellyzly, please provide the privilege to me to update
The version that will support Spark2.0 is Hive2.2
No not know yet when this is going to be release.
-Original Message-
From: baipeng [mailto:b...@meitu.com]
Sent: 07 December 2016 08:04
To: user@hive.apache.org
Subject: When Hive on Spark will support Spark 2.0?
Does Anyone know when
Does Anyone know when Hive will release version to support Spark 2.0? Now hive
2.1.0 only supports spark 1.6.
Being unable to integrate separately Hive with Spark I just started directly on
Spark the thrift server.
Now it is working as expected.
From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com]
Sent: 29 November 2016 11:12
To: user <user@hive.apache.org>
Subject: Re: Hive on Spark not w
HI Mich
I read in some older post that you make it work as well with the configuration
I have:
Hive2.0.1--> Spark 1.6.2 –> Hadoop – 2.6.5 --> Scala 2.10
You only make it work with Hive 1.2.1 --> Spark 1.3.1 --> etc ….?
BR
Joaquin
From: Mich Talebzadeh [mailto:mich.talebza...@
Hive on Spark engine only works with Spark 1.3.1.
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpre
ClassNotFoundException generally means that jars are missing from your
class path.
You probably need to link the spark jar to $HIVE_HOME/lib
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive
On Tue, Nov 29, 2016 at 2:03
the search directly through spark-shell it work great.
+---+
|_c0|
+---+
|1005635|
+---+
So there has to be a problem from hive to spark.
Seems as the RPC(??) connection is not setup Can somebody guide me on what
to look for.
spark.master=spark://172.16.173.31:7077
hive.execution.en
all...
>
> John
>
>
> On Thu, Sep 15, 2016 at 4:20 PM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Yes you can. Hive on Spark meaning Hive using Spark as its execution
>> engine works fine. The version that I managed to make it work is any
Did you run it on Mesos? Your presentation doesn't mention Mesos at all...
John
On Thu, Sep 15, 2016 at 4:20 PM, Mich Talebzadeh <mich.talebza...@gmail.com>
wrote:
> Yes you can. Hive on Spark meaning Hive using Spark as its execution
> engine works fine. The version that I man
Hey all, I was experimenting with some bleeding edge Hive. (2.1) and
trying to get it to run on bleeding edge Spark (2.0).
Spark is working fine, I can query the data all is setup, however, I can't
get Hive on Spark to work. I understand it's not really a thing (Hive on
Spark on Mesos) but I am
?
>
> I cannot see spark.home in your hive-site.xml so I cannot figure it out.
>
> BTW you are using Spark standalone as the mode. I tend to use yarn-client.
>
> Now back to the above issue. Do other queries work OK with Hive on Spark?
>
> Some of those perf parameters can be set u
with Hive on Spark?
Some of those perf parameters can be set up in Hive session itself or
through init file
set spark.home=/usr/lib/spark-1.6.2-bin-hadoop2.6;
set spark.master=yarn;
set spark.deploy.mode=client;
set spark.executor.memory=8g;
set spark.driver.memory=8g;
set spark.executor.instances=6
Hi,
After several days trying to figure out the problem I'm stuck with a class
cast exception when running a query with hive on spark on orc tables that I
updated with the streaming mutation api of hive 2.0.
The context is the following:
For hive:
The version is the latest available from
All the parameters except spark.executor.instances are specified in
spark-default.conf located in hive's conf folder. So I think it's a yes.
I also checked on spark's web page when a hive on spark job is running, the
parameters shown on the web page are exactly what I specified in the config
when you start hive on spark do you set any parameters for the submitted
job (or read them from init file)?
set spark.master=yarn;
set spark.deploy.mode=client;
set spark.executor.memory=3g;
set spark.driver.memory=3g;
set spark.executor.instances=2;
set spark.ui.port=;
Dr Mich Talebzadeh
Hi Mich!
There is no problem is displaying records or performing any aggregations on
the records after inserting data from spark into the hive table. It is the
count query (in hive) that returns the wrong result in hive prior to
issuing the compute statistics command.
On Mon, Aug 22, 2016 at 4
from
such loss, damage or destruction.
On 22 August 2016 at 11:51, Nitin Kumar <nk94.nitinku...@gmail.com> wrote:
> Hi Furcy,
>
> If I execute the command "ANALYZE TABLE TEST_ORC COMPUTE STATISTICS"
> before checking the count from hive, Hive returns the correct count albeit
&g
Hi Furcy,
If I execute the command "ANALYZE TABLE TEST_ORC COMPUTE STATISTICS" before
checking the count from hive, Hive returns the correct count albeit it does
not spawn a map-reduce job for computing the count.
I'm running a HDP 2.4 Cluster with Hive 1.2.1.2.4 and Spark 1.6.1
If
Nitin Kumar <nk94.nitinku...@gmail.com>
wrote:
> Hi!
>
> I've noticed that hive has problems in registering new data records if the
> same table is written to using both the hive terminal and spark sql. The
> problem is demonstrated throug
Hi!
I've noticed that hive has problems in registering new data records if the
same table is written to using both the hive terminal and spark sql. The
problem is demonstrated through the commands listed below
hive> use defa
; explained how to make Spark work as an execution engine for Hive.
>
> Query Engines for Hive, MR, Spark, Tez and LLAP – Considerations
> <http://www.meetup.com/futureofdata-london/events/232423292/>!
>
> See if I can send the presentation
>
> C
tation in London on 20th July on this subject:. In that I
> explained how to make Spark work as an execution engine for Hive.
>
> Query Engines for Hive, MR, Spark, Tez and LLAP – Considerations!
>
> See if I can send the presentation
>
> Cheers
>
>
> Dr Mich
Thanks Guys for the help!
Thanks,
Mudit
From: Mich Talebzadeh <mich.talebza...@gmail.com>
Reply-To: <user@hive.apache.org>
Date: Thursday, July 28, 2016 at 9:43 AM
To: user <user@hive.apache.org>
Subject: Re: Hive on spark
Hi,
I made a presentation in London on 20th Ju
Hi,
I made a presentation in London on 20th July on this subject:. In that I
explained how to make Spark work as an execution engine for Hive.
Query Engines for Hive, MR, Spark, Tez and LLAP – Considerations
<http://www.meetup.com/futureofdata-london/events/232423292/>!
See if I ca
mudit,
this link can guide you -
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
Thanks,
Karthik
On Thu, Jul 28, 2016 at 8:54 AM, Mudit Kumar <mudit.ku...@askme.in> wrote:
> Yes Mich,exactly.
>
> Thanks,
> Mudit
>
> From: Mich
Yes Mich,exactly.
Thanks,
Mudit
From: Mich Talebzadeh <mich.talebza...@gmail.com>
Reply-To: <user@hive.apache.org>
Date: Thursday, July 28, 2016 at 1:08 AM
To: user <user@hive.apache.org>
Subject: Re: Hive on spark
You mean you want to run Hive using Spark as the exec
You mean you want to run Hive using Spark as the execution engine which
uses Yarn by default?
Something like below
hive> select max(id) from oraclehadoop.dummy_parquet;
Starting Spark Job = 8218859d-1d7c-419c-adc7-4de175c3ca6d
Query Hive on Spark job[1] stages:
2
3
Status: Running (H
Hi All,
I need to configure hive cluster based on spark engine (yarn).
I already have a running hadoop cluster.
Can someone point me to relevant documentation?
TIA.
Thanks,
Mudit
.@gmail.com>
Cc: user @spark <u...@spark.apache.org>; user <user@hive.apache.org>
Subject: Re: Presentation in London: Running Spark on Hive or Hive on Spark Dr
Mich do you have any slides or videos available for the presentation you did
@Canary Wharf?kindest regards marco On Wed,
co Mistroni [mailto:mmistr...@gmail.com]
> *Sent:* 15 July 2016 11:04
> *To:* Mich Talebzadeh <mich.talebza...@gmail.com>
> *Cc:* user @spark <u...@spark.apache.org>; user <user@hive.apache.org>
> *Subject:* Re: Presentation in London: Running Spark on Hive or H
Hi Mich,
Would it be on YouTube , post session ?
- Harmeet
On Jul 7, 2016, at 3:07, Mich Talebzadeh <mich.talebza...@gmail.com> wrote:
> Dear forum members
>
> I will be presenting on the topic of "Running Spark on Hive or Hive on Spark,
> your mileage varies&quo
Thanks.
Will this presentation recorded as well?
Regards
On Wednesday, 6 July 2016, 22:38, Mich Talebzadeh
<mich.talebza...@gmail.com> wrote:
Dear forum members
I will be presenting on the topic of "Running Spark on Hive or Hive on Spark,
your mileage varies" in Future
Dear forum members
I will be presenting on the topic of "Running Spark on Hive or Hive on
Spark, your mileage varies" in Future of Data: London
<http://www.meetup.com/futureofdata-london/events/232423292/>
*Details*
*Organized by: Hortonworks <http://hortonworks.com/>*
Hi
we are using Cloudera 5.7.0
there's a use case to process XML data,
we are using the https://github.com/dvasilen/Hive-XML-SerDe
XML serde is working with Hive execution engine as Map-Reduce,
we enabled Hive on Spark to test the performance, and we are facing
following issue
16/06/23 12
Thanks Ted,
More interested in general availability of Hive 2 on Spark 1.6 engine as
opposed to Vendors specific custom built.
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view
Thanks Jorn.
Just to be clear they get Hive working with Spark 1.6 out of the box
(binary download)? The usual work-around is to build your own package and
get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
Cheers
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile
g the binary downloads. I
> just tried Hive 2 on Spark 1.6 as the execution engine and it crashed.
>
> I do not know the development state of this cross-breed but will be very
> desirable if we could manage to sort out this
> spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>
1 - 100 of 313 matches
Mail list logo
