Re: Specifying YARN Node (Label) for LLAP AM

am closing my argument. Cheers Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any a

Re: Specifying YARN Node (Label) for LLAP AM

luding spark thrift server (which is under the bonnet Hive thrift server). HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh

Re: Specifying YARN Node (Label) for LLAP AM

Hi, Are you using LLAP (Long live and prosper) as a Hive engine? HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Tale

Re: Hive 3 has big performance improvement from my test

on this so please clarify the above statement HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of d

Re: Hive 3 has big performance improvement from my test

us to know this. Thanks view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property whic

Re: Hive unable to Launch job to spark

iew my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on

Insert from Hive table into Google BigQuery table

155_9c3bc173-6893-4a5b-a061-226e2ac9b42a Time taken: 2.558 seconds 2022-03-08 15:51:58,296 INFO [ec1f7b38-7286-481a-b48b-44f7ed54c20a main] CliDriver: Time taken: 2.558 seconds 2022-03-08 15:51:58,296 INFO [ec1f7b38-7286-481a-b48b-44f7ed54c20a main] conf.HiveConf: Using the default value passed in for

Re: help with beeline connection to hive

and check that beeline thrift server is indeed running (mine runs on port 10099) netstat -plten|grep 10099 tcp0 0 0.0.0.0:10099 0.0.0.0:* LISTEN 1 view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>

Re: help with beeline connection to hive

beeline -u jdbc:hive2://localhost:1/default org.apache.hive.jdbc.HiveDriver -n -p view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsi

Re: metastore bug when hive update spark table ?

IgnoreKeyTextOutputFormat', comment='') Row(col_name='Storage Properties', data_type='[serialization.format=1]', comment='') Row(col_name='Partition Provider', data_type='Catalog', comment='') This is my work around HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-520

Re: [ANNOUNCE] Apache Hive 2.3.9 Released

ive. Looking forward to seeing many happy years with Apache Hive. view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any o

Re: Is a Hive installation necessary for Spark SQL?

rl", url). \ option("dbtable", tableName). \ option("user", user). \ option("password", password). \ option("driver", driver). \ mode(mode). \ save() except Exception as e:

Re: [BUG] Hive 3.1.2 ALTER TABLE statement

tps://stackoverflow.com/questions/2075/hive-execution-error-return-code-1-from-org-apache-hadoop-hive-ql-exec-ddltask> HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility

Re: [BUG] Hive 3.1.2 ALTER TABLE statement

. . . . . . . . . . . . . . . . . . > No rows affected (0.002 seconds) HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise fro

Re: Migration of Hadoop Warehouse to newer versions lead to bad performance for JDBC-data-retrieval

ng Director: Dr. Thilo Gans, Bernd Vermaaten > Webseite | www.solute.de > Sitz | Registered Office: Karlsruhe > Registergericht | Register Court: Amtsgericht Mannheim > Registernummer | Register No.: HRB 110579 > USt-ID | VAT ID: DE234663798 > > *Informationen zum Datenschutz | Inform

Re: Migration of Hadoop Warehouse to newer versions lead to bad performance for JDBC-data-retrieval

Hi Juuien, I assume you mean you are using JDBC drivers to retrieve from the source table in Hive (older version) to the target table in Hive (newer version). 1) what JDBC drivers are you using? 2) Are these environments kerberized in both cases? 3) Have you considered other JDBC drivers for

Re: Is Insert Overwrite table partition on s3 is an atomic operation ?

Hi Mark, By atomic operation I gather you mean INSERT/OVERWRITE affects that partition only? According to my somehow dated scripts yes you can do that. The idea being that you only want to overwrite data for that partition ONLY. --show create table marketData; --Populate target table select

Re: How useful are tools for Hive data modeling

tools question, but mentioning it for > completeness… > > Thanks > > Austin > > > On 11 Nov 2020, at 17:14, Mich Talebzadeh > wrote: > > Many thanks Peter. > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw &g

Re: How useful are tools for Hive data modeling

; Hi Mich, > > Index support was removed from hive: > >- https://issues.apache.org/jira/browse/HIVE-21968 >- https://issues.apache.org/jira/browse/HIVE-18715 > > > Thanks, > Peter > > On Nov 11, 2020, at 17:25, Mich Talebzadeh > wrote: > > Hi al

Fwd: How useful are tools for Hive data modeling

Hi all, I wrote these notes earlier this year. I heard today that someone mentioned Hive 1 does not support indexes but hive 2 does. I still believe that Hive does not support indexing as per below. Has this been changed? Regards, Mich -- Forwarded message - From: Mich

Re: what does MM stand for

MM -> Micro Managed Hive managed tables supporting Insert-only operations with ACID semantics are called MM (Micro-Managed) OR Insert-Only ACID tables. Supports all file formats. >From say

Re: SQL CTAS query failed on compilation stage

/HIVE-24352 > > Regards, > Bartosz Kotwica > > wt., 3 lis 2020 o 11:57 Mich Talebzadeh > napisał(a): > >> well you have to be pragmatic. That may well be a bug due to Hive, >> especially it says "Also check for circular dependencies" >> >> yo

Re: SQL CTAS query failed on compilation stage

s up when CTE or subquery in from clause is used. > > wt., 3 lis 2020 o 10:45 Mich Talebzadeh > napisał(a): > >> Hm, >> >> Hi Bartosz, >> >> Can you create a temporary table with your sub-query and see it works? >> >> create temporary table ta

Re: SQL CTAS query failed on compilation stage

Hm, Hi Bartosz, Can you create a temporary table with your sub-query and see it works? create temporary table tab2 as ... HTH LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Hive using Spark engine vs native spark with hive integration.

Hi Manu, In the past (July 2016), I made a presentation organised by then Hortonworks in London titled "Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations! " The PDF presentation is here . With a caveat

Re: Measuring the execution time of Hive queries through Ambari

ht Mannheim > Registernummer | Register No.: HRB 110579 > USt-ID | VAT ID: DE234663798 > > *Informationen zum Datenschutz | Information about privacy policy* > https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php > > > > --

Re: Measuring the execution time of Hive queries through Ambari

USt-ID | VAT ID: DE234663798 > > *Informationen zum Datenschutz | Information about privacy policy* > https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php > > > > -- > *Von:* Mich Talebzadeh > *Gesendet:* Montag, 22. Juni 2020 12:

Measuring the execution time of Hive queries through Ambari

Hi, Using Ambari to connect to Hive, is there any way of measuring the query time? Please be aware that this is through Ambari not through beeline etc. The tool we have at the moment is Ambari to Prod. We do not have any other luxury! Thanks LinkedIn *

Re: Running Hive queries from Ambari or from edge node via beeline

T ID: DE234663798 > > *Informationen zum Datenschutz | Information about privacy policy* > https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php > > > > -- > *Von:* Mich Talebzadeh > *Gesendet:* Montag, 15. Juni 2020 21:26:20 > *An:

Running Hive queries from Ambari or from edge node via beeline

Hi, I am not a user of Ambari but I believe it is a GUI interface to Hive. it can be run on your laptop and connect to Hive via ODBC or JDBC. There is also another tool DB Visualizer Pro that uses JDBC to connect to Hive thrift server. My view is that if one is a developer the best best would

Re: create transactional table issue

In Hive 3.1.1 the thread owner table creation works fine hive --version Hive 3.1.1 0: jdbc:hive2://rhes75:10099/default> create table test.dllm ( b string ) partitioned by (a int) clustered by (b) into 2 buckets stored as orc tblproperties('transactional'='true') . . . . . . . . . . . . . . . .

Re: Adding a virtual column for a custom input format

Hi Christine. Virtual column meaning a derived column? can you achieve this by creating a view on Hive table? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

How useful are tools for Hive data modeling

because Hadoop lacks blocks locality necessary for indexes. So If I use a tool like Collibra, Ab-intio etc what advantage(s) one is going to gain on top a simple sell scrip to get table and partition definitions? Thanks, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Hive external table not working in sparkSQL when subdirectories are present

Have you updated partition statistics by any chance? I assume you can access the table and data though Hive itself? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

Re: Hive external table not working in sparkSQL when subdirectories are present

which versions of Spark and Hive are you using. what will happen if you use parquet tables instead? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

Re: Error: java.io.IOException: java.lang.RuntimeException: ORC split generation failed with exception: java.lang.NoSuchMethodError

with protoc 2.5.0 >From source with checksum 14182d20c972b3e2105580a1ad6990 It was installed a year back. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/v

Error: java.io.IOException: java.lang.RuntimeException: ORC split generation failed with exception: java.lang.NoSuchMethodError

Just upgraded Hive from Hive-3.0 to 3.1.1 Connected to: Apache Hive (version 3.1.1) Driver: Hive JDBC (version 3.1.1) Created an ORC table through Spark as below: sql("use accounts") // // Drop and create table ll_18740868 // sql("DROP TABLE IF EXISTS accounts.ll_18740868") var sqltext = ""

Re: hcatalog and hiveserver2

er.fpp"="0.05", "orc.compress"="SNAPPY", "orc.stripe.size"="16777216", "orc.row.index.stride"="1" ) """ HiveContext.sql(sqltext) // // Put data in Hive table. Clean up is already done

Re: Any HIVE DDL statement takes minutes to execute

Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Re: Hive metastore service

Try this Assuming that you are talking about Hive Thrift server beeline -u jdbc:hive2://rhes75:10099/default org.apache.hive.jdbc.HiveDriver *-n USERNAME -p PASSWORD* -i /home/hduser/dba/bin/add_jars.hql' HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Comparing Google Cloud Platform BiqQuery with Hive

I have founds out that using Spark in both prem and GCP on Hive and BQ respectively makes things easier. Also so far as my tests go Spark has analytical functions identical both on prem and in Dataproc. HTH, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Comparing Google Cloud Platform BiqQuery with Hive

ave not tried it myself). So the question is are there any advantages taking a Hive table into BQ itself? Thanks, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/v

Re: Trying to create an extrenal table in Hive for MongoDBthrows error

accounts.ll_18740868_mongo; You will need to load the jar files whenever you need to access this table In MongoDB I have > use accounts; switched to db accounts > db.ll_18740868.count() 3623 And confirmed in Hive 0: jdbc:hive2://rhes75:10099/default> select count(1) from a

Trying to create an extrenal table in Hive for MongoDBthrows error

23:01:32,447 INFO [main] session.SessionState: Deleted directory: /tmp/hive/hduser/a928186b-244d-4b81-bda9-dd3a83e306b2 on fs with scheme hdfs 2018-08-26 23:01:32,447 INFO [main] session.SessionState: Deleted directory: /tmp/hive/a928186b-244d-4b81-bda9-dd3a83e306b2 on fs with scheme file 2018-0

Re: Hive Metada as a microservice

Thanks I believe we can classify it as a microservice as it provides metadata service for various other artefacts and it falls into the definition of loosely coupled service. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Hive Metada as a microservice

Hi, My understanding is that in later releases of Hive, the metadata will be a separate offerings. Will this be a type of microservice offering providing loose coupling to various other artefact? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Error Starting hive thrift server, hive 3 on Hadoop 3.1

Sure will do Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at yo

Re: Error Starting hive thrift server, hive 3 on Hadoop 3.1

artefacts and results in unnecessary waste of time. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Error Starting hive thrift server, hive 3 on Hadoop 3.1

/Configuration;)Lorg/apache/htrace/HTraceConfiguration; Any ideas? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Re: Reading XML into Hive table, handing null columns

namenode | NULL | +---+++----+ HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Discla

Reading XML into Hive table, handing null columns

ext()') as description from xml_temp; +--+ | description | +--+ | [] | | [] | | [] | | [] | | [] | | [] | | [] | | [] | | [] | +--+ Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http:/

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Re: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

sum 736cdcefa911261ad56d2d120bf1fa This command was run using /home/hduser/hadoop-3.0.3/share/hadoop/common/hadoop-common-3.0.3.jar Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/v

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split generation failed with exception

ion: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I (state=08S01,code=1) Something is missing here! Is this specific to ORC tables? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <

Update on ORC transactional table fails with org.apache.hadoop.fs.FileStatus.compareTo.. error

ive.ql.exec.StatsTask. org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I (state=08S01,code=-101) Appreciate any info. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.co

Hive 3 Thrift server expects Tez to be there and throws error and waits before bouncing back

Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Re: Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

=${HADOOP_HOME} mapreduce.reduce.env HADOOP_MAPRED_HOME=${HADOOP_HOME} Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Hive 3,0 on Hadoop 3.0.3 crahes with org.apache.hadoop.mapreduce.v2.app.MRAppMaster error

6_02_01/stderr " Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your

Re: Which version of Hive can hanle creating XML table?

many thanks. but I cannot see any specific product name there? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Re: May 2018 Hive User Group Meeting

yes indeed I second that Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer

reading xml file with xpath into hive table it expects the xml without white space and carriage return?

| _c0 | _c1 | _c2 | _c3 | _c4|_c5 |_c6 |_c7 | +---+-----+-----+----+--++++ | Mich | ["

Re: Which version of Hive can hanle creating XML table?

mldata,'employee/designation') FROM xml_table_org;* Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.c

Re: Which version of Hive can hanle creating XML table?

Ok I am looking at this jar file jar tf hive-serde-3.0.0.jar|grep -i abstractserde org/apache/hadoop/hive/serde2/AbstractSerDe.class Is this the correct one? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <ht

Re: Which version of Hive can hanle creating XML table?

Thanks Jorn so what is the resolution? do I need another jar file? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Re: Which version of Hive can hanle creating XML table?

nputFormat.class com/ibm/spss/hive/serde2/xml/XmlSerDe$1.class com/ibm/spss/hive/serde2/xml/XmlSerDe.class META-INF/maven/ META-INF/maven/com.ibm.spss.hive.serde2.xml/ META-INF/maven/com.ibm.spss.hive.serde2.xml/hivexmlserde/ META-INF/maven/com.ibm.spss.hive.serde2.xml/hivexmlserde/pom.xml META-INF/m

Re: issues with Hive 3 simple sellect from an ORC table

Hi Owen, It is 2.7.3 hadoop version Hadoop 2.7.3 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff Compiled by root on 2016-08-18T01:41Z Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Which version of Hive can hanle creating XML table?

org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe (state=08S01,code=1) Does anyone know the cause of this or which version of Hive supports creating an XML table? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/

issues with Hive 3 simple sellect from an ORC table

| 1| +---++--+ 48 rows selected (0.561 seconds) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOAB

Re: Oracle 11g Hive 2.1 metastore backend

) HTH, Mich Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own ris

Re: Cannot send metadata info from Hive 2.0.1 to Hive metastore on Oracle 12c

omment | +---++--+ | col1 | int| | +---++--+ 1 row selected (0.147 seconds) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profil

Cannot send metadata info from Hive 2.0.1 to Hive metastore on Oracle 12c

n; VER_ID, SCHEMA_VERSION, VERSION_COMMENT 1 2.0.0 Hive release version 2.0.0 Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

What has changed in Hive 2.3.2 that cannot use Spark engine.

ideas what has changed? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it a

Re: Query failing in Hive 2.2

it is compatible with what is stored in the old partitions. That I believe will work. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Hive 2.3.2 does not execute on Spark engine anymore

TEZ LLAP at this stage. Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer

Re: partitioned hive table

have you analyzed table for the partition? ANALYZE TABLE test_table PARTITION('2017-08-20, bar='hello'') COMPUTE STATISTICS; and do count(*) from table Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.

Can one classify Hive as an analytical tool besides storage?

supports HQL that in turn has analytical functions like RANK etc built in. So in effect it is not only a storage, but can be used as an analytical tool as well? What are your views? Thanks, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Hive query on ORC table is really slow compared to Presto

With ORC tables have you tried set hive.vectorized.execution.enabled = true; set hive.vectorized.execution.reduce.enabled = true; SET hive.exec.parallel=true; -- set hive.optimize.ppd=true; HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: setup spark engine to hive ,the hive version and spark build problem

just to clarify you mean: Hive 2.1.1 works with Spark engine 2.X Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Re: setup spark engine to hive ,the hive version and spark build problem

NFO : Starting task [Stage-1:MAPRED] in serial mode INFO : *Query Hive on Spark job[0] stages:* INFO : 0 INFO : 1 INFO : *Status: Running (Hive on Spark job[0])* HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <

Re: Pro and Cons of using HBase table as an external table in HIVE

BLE MARKETDATAHBASE (PK VARCHAR PRIMARY KEY, PRICE_INFO.TICKER VARCHAR, PRICE_INFO.TIMECREATED VARCHAR, PRICE_INFO.PRICE VARCHAR); HTH, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAE

Re: Jimmy Xiang now a Hive PMC member

Best wishes in the new role Jimmy. Regards, Mich Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Hive handling of ingested data when source column changes size or new column added

, then there is no other way than adding that column to Hive. Also the existing Hive partitions will stay as before but new partitions will have space reserved for additional columns. Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Unable to retrieve table metadata from hcatalog

yes I do it is there Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer

Unable to retrieve table metadata from hcatalog

) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1228) at org.apache.hive.hcatalog.api.HCatClientHMSImpl.getTable(HCatClientHMSImpl.java:168) I am not that familiar with HCat. Any help will be appreciated. thanks Dr Mich Talebzadeh LinkedIn * https

Re: beeline connection to Hive using both Kerberos and LDAP with SSL

So it translates to either LDAP or Kerberos, we cannot enable both for same Hive Server. SSL is independent. So the supported situations are as below. 1. Anonymous authentication (w/ or w/o SSL) 2. LDAP authentication (w/ or w/o SSL) 3. Kerberos Cheers Dr Mich Talebzadeh

Re: beeline connection to Hive using both Kerberos and LDAP with SSL

Thanks Kapil. Does this mean that one can have both Kerberos and LDAP (with SSL) and use either? Cheers, Mich Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

beeline connection to Hive using both Kerberos and LDAP with SSL

Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsi

Real time data streaming into Hive text table and excessive amount of file numbers

files? Is there any detriment when the number of these files grow very high such as 1000s of them? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6Ac

Re: Using Sqoop to get data from Impala/Hive to another Hive table

regardless there is no point using Sqoop for such purpose. it is not really designed for it :) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV

Re: Using Sqoop to get data from Impala/Hive to another Hive table

. this is not really a test is iut? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer

Using Sqoop to get data from Impala/Hive to another Hive table

Hi, I have not tried this but someone mentioned that it is possible to use Sqoop to get data from one Impala/Hive table in one cluster to another? The clusters are in different zones. This is to test the cluster. Has anyone done such a thing? Thanks Dr Mich Talebzadeh LinkedIn * https

Re: Difference between join and inner join

join is by default inner join as in Oracle or Sybase. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Parquet tables with snappy compression

Hi, Has there been any study of how much compressing Hive Parquet tables with snappy reduces storage space or simply the table size in quantitative terms? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <ht

Re: VARCHAR or STRING fields in Hive

Sounds like VARCHAR and CHAR types were created for Hive to have ANSI SQL Compliance. Otherwise they seem to be practically the same as String types. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.

Re: VARCHAR or STRING fields in Hive

as opposed to String make any difference in terms of storage efficiency? Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV

Re: VARCHAR or STRING fields in Hive

in HDFS compared to STRING columns? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disc

VARCHAR or STRING fields in Hive

to STRRING. What is the thread view on this? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disc

Vectorised Queries in Hive

confirms this please? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at yo

Re: Specifying orc.stripe.size in Spark

.filter.columns"="ID", "orc.bloom.filter.fpp"="0.05", "orc.stripe.size"="268435456", "orc.row.index.stride"="1" ) """ HiveContext.sql(sqltext) sqltext = """ INSERT INTO TABLE test.dummy2 SELECT

  1   2   3   4   5   6   7   8   >