Thanks Gopal for your inputs ....For now I have create NON ACID table and
loaded data ....see below from logs proper group splits happening .
2016-06-25 12:52:00,160 [INFO] [InputInitializer {Map 1} #0]
|tez.HiveSplitGenerator|: Number of grouped splits: 512
On compaction issue , Compaction enabled with two workers. why compaction
not happened ? will check metastore logs.
I have too many ACID tables on hive and how many worker should be
configured ? currently it is 2.
Thanks a lot once again.
Regards
Sanjiv Singh
Mob : +091 9990-447-339
On Fri, Jun 24, 2016 at 9:14 PM, @Sanjiv Singh <sanjiv.is...@gmail.com>
wrote:
> Thanks Gopal for your inputs. Let me run compaction explicitly on table
> then see how query works.
>
>
>
> Let
>
> Regards
> Sanjiv Singh
> Mob : +091 9990-447-339
>
> On Fri, Jun 24, 2016 at 7:53 PM, Gopal Vijayaraghavan <gop...@apache.org>
> wrote:
>
>>
>> > Yes for this tables, ACID enabled. it has only 256 files for each
>> >buckets. these are create only when data initially loaded in this table.
>>
>> Yes, the initial load goes in as an insert DELTA too - that requires
>> another compaction to move into base files.
>>
>> The fact that they haven't been automatically compacted yet, suggests that
>> the compactor isn't working for some reason (check hive metastore logs).
>>
>> > One thing that I am not able to understand that its is running with 1
>> >MAPPER.
>>
>> The size of deltas shows up as 0, till the compaction goes through - in
>> Hive2, it will be -1 which will be correctly interpreted as "unknown
>> size".
>>
>>
>> > | -rw-r--r-- 3 H56473 hdfs 215973009 2016-06-23 17:38
>>
>> >/apps/hive/warehouse/PRDDB.db/tuning_dd_key/delta_0001570_0001570/bucket_0
>> >0000 |
>>
>> Clearly an issue due to the lack of compaction - I see a single delta with
>> 255 buckets and no base_* files at all.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
