I also thought about it, but no, it's not consistency.

Consistency view is enabled. I use same s3 for my own map-reduce jobs and
it's ok.

I also checked if it lost consistency (emrfs diff). No problems.

In case of inconsistency of s3 files disappear right after they were
written and appear some time after. Hfiles didn't appear after a day, but
_template is there.

It's 100% reproducable, I think I'll investigate this problem by running
conversion job manually.

10 авг. 2017 г. 17:18 пользователь "ShaoFeng Shi" <shaofeng...@apache.org>
написал:

> Did you enable the Consistent View? This article explains the challenge
> when using S3 directly for ETL process:
> https://aws.amazon.com/cn/blogs/big-data/ensuring-
> consistency-when-using-amazon-s3-and-amazon-elastic-
> mapreduce-for-etl-workflows/
>
>
> 2017-08-09 18:19 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>
>> Yes, it's empty. Also I see this message in the log:
>>
>> 2017-08-09 09:02:35,947 WARN  [Job 1e436685-7102-4621-a4cb-6472b866126d-7608]
>> mapreduce.LoadIncrementalHFiles:234 : Skipping non-directory
>> s3://joom.emr.fs/home/production/bi/kylin/kylin_metadata/
>> kylin-1e436685-7102-4621-a4cb-6472b866126d
>> /main_event_1_main/hfile/_SUCCESS
>> 2017-08-09 09:02:36,009 WARN  [Job 1e436685-7102-4621-a4cb-6472b866126d-7608]
>> mapreduce.LoadIncrementalHFiles:252 : Skipping non-file
>> FileStatusExt{path=s3://joom.emr.fs/home/production/bi/kylin
>> /kylin_metadata/kylin-1e436685-7102-4621-a4cb-6472b866126d/
>> main_event_1_main/hfile/_temporary/1; isDirectory=true;
>> modification_time=0; access_time=0; owner=; group=; permission=rwxrwxrwx;
>> isSymlink=false}
>> 2017-08-09 09:02:36,014 WARN  [Job 1e436685-7102-4621-a4cb-6472b866126d-7608]
>> mapreduce.LoadIncrementalHFiles:422 : Bulk load operation did not find
>> any files to load in directory s3://joom.emr.fs/home/producti
>> on/bi/kylin/kylin_metadata/kylin-1e436685-7102-4621-a4cb-
>> 6472b866126d/main_event_1_main/hfile.  Does it contain files in
>> subdirectories that correspond to column family names?
>>
>> On Wed, Aug 9, 2017 at 1:15 PM, ShaoFeng Shi <shaofeng...@apache.org>
>> wrote:
>>
>>> The HFile will be moved to HBase data folder when bulk load finished;
>>> Did you check whether the HTable has data?
>>>
>>> 2017-08-09 17:54 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>>>
>>>> Hi!
>>>>
>>>> I set kylin.hbase.cluster.fs to s3 bucket where hbase lives.
>>>>
>>>> Step "Convert Cuboid Data to HFile" finished without errors.
>>>> Statistics at the end of the job said that it has written lot's of data to
>>>> s3.
>>>>
>>>> But there is no hfiles in kylin_metadata folder (kylin_metadata
>>>> /kylin-1e436685-7102-4621-a4cb-6472b866126d/<table name>/hfile), but
>>>> only _temporary folder and _SUCCESS file.
>>>>
>>>> _temporary contains hfiles inside attempt folders. it looks like there
>>>> were not copied from _temporary to result dir. But there is no errors
>>>> neither in kylin log, nor in reducers' logs.
>>>>
>>>> Then loading empty hfiles produces empty segments.
>>>>
>>>> Is that a bug or I'm doing something wrong?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

Reply via email to