Jorn, This is kind of one time load from Historical Data to Analytical Hive engine. Hive version 1.2.1 and Spark version 2.0.1 with MapR distribution.
Writing every table to parquet and reading it could be very much time consuming, currently entire job could take ~8 hours on 8 node of 100 Gig ram 20 core cluster, not only used utilized by me but by larger team. Thanks On Fri, Nov 3, 2017 at 1:31 AM, Jörn Franke <jornfra...@gmail.com> wrote: > Hi, > > Do you have a more detailed log/error message? > Also, can you please provide us details on the tables (no of rows, > columns, size etc). > Is this just a one time thing or something regular? > If it is a one time thing then I would tend more towards putting each > table in HDFS (parquet or ORC) and then join them. > What is the Hive and Spark version? > > Best regards > > > On 2. Nov 2017, at 20:57, Chetan Khatri <chetan.opensou...@gmail.com> > wrote: > > > > Hello Spark Developers, > > > > I have 3 tables that i am reading from HBase and wants to do join > transformation and save to Hive Parquet external table. Currently my join > is failing with container failed error. > > > > 1. Read table A from Hbase with ~17 billion records. > > 2. repartition on primary key of table A > > 3. create temp view of table A Dataframe > > 4. Read table B from HBase with ~4 billion records > > 5. repartition on primary key of table B > > 6. create temp view of table B Dataframe > > 7. Join both view of A and B and create Dataframe C > > 8. Join Dataframe C with table D > > 9. coleance(20) to reduce number of file creation on already > repartitioned DF. > > 10. Finally store to external hive table with partition by skey. > > > > Any Suggestion or resources you come across please do share suggestions > on this to optimize this. > > > > Thanks > > Chetan >
