Problems with Hive Streaming. Compactions not working. Out of memory errors.

Hi all, We are trying to use Hive streaming to ingest data in real time from Flink. We send batches of data every 5 seconds to Hive. We are working version 1.1.0-cdh5.8.2. The ingestión works fine. However, compactions are not working, the log shows this error: Unable to select next element

RE: Hive on Spark not working

Being unable to integrate separately Hive with Spark I just started directly on Spark the thrift server. Now it is working as expected. From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] Sent: 29 November 2016 11:12 To: user Subject: Re: Hive on Spark not working

Re: Problems with Hive Streaming. Compactions not working. Out of memory errors.

The OOM is most likely a side effect of not running compactions. W/o compactions you never reduce the number of delta file that need to be loaded to materialize the data set on read. On 11/29/16, 10:03 AM, "Alan Gates" wrote: >I¹m guessing that this is an issue in the

Reg:Sqoop Import-Oracle to Hive-Parquet

Hi Experts, We are trying to use parquet for importing data from oracle to hive, we are encountering the below error, could anyone help me to resolve this issue ? We are using sqoop version 1.4.6 and hive version 1.2. Error: 16/11/28 21:21:46 INFO hive.metastore: Connected to

Re: Issues regarding HPLSQL tool

Ainhoa, Can you please post the entire script? Also try to add --trace option to see which query was actually executed in the database. Thanks, Dmitry On Tue, Nov 29, 2016 at 11:44 AM, Ainhoa Benitez wrote: > Good morning, > > my name is Ainhoa. I am starting to use

Re: Hive on Spark not working

Hive on Spark engine only works with Spark 1.3.1. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com

Re: Issues regarding HPLSQL tool

Ainhoa, Did you provide correct username and password to connect to Hive server? The problem I see is that it cannot launch a MapReduce job. SELECT * FROM tab does not require a MR job so it works fine, while when you add WHERE clause it fails. Thanks, Dmitry On Tue, Nov 29, 2016 at 2:33 PM,

RE: Hive on Spark not working

HI Mich I read in some older post that you make it work as well with the configuration I have: Hive2.0.1--> Spark 1.6.2 –> Hadoop – 2.6.5 --> Scala 2.10 You only make it work with Hive 1.2.1 --> Spark 1.3.1 --> etc ….? BR Joaquin From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] Sent:

RE: Need error logging advice on batch processing

Usually if DDL fails, query is displayed one line above line with keyword Failed bash> cat CTASchybne2.log | grep -n -B 1 FAILED 4-create table SCHEMA.AAA stored as orc as select * from SCHEMA.AAA 5:FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Table already

Re: Question about partition pruning when there's a type mismatch

Thanks for the tips, Gopal. I stepped through the code in a debugger and found that in the case of String = String, the predicate was pushed down to the SQL query on the metastore side, whereas in the case of String = Int, the SQL filter pushdown failed, so GenericUDFOPEqual gets evaluated and

Re: Problems with Hive Streaming. Compactions not working. Out of memory errors.

I’m guessing that this is an issue in the metastore database where it is unable to read from the transaction tables due to the ingestion rate. What version of Hive are you using? What database are you storing the metadata in? Alan. > On Nov 29, 2016, at 00:05, Diego Fustes Villadóniga