Any ideas how to fix that?

On Fri, Aug 11, 2017 at 2:16 PM, ShaoFeng Shi <shaofeng...@apache.org>
wrote:

> I got the same problem as you:
>
> 2017-08-11 08:44:16,342 WARN  [Job 2c86b4b6-7639-4a97-ba63-63c9dca095f6-2255]
> mapreduce.LoadIncrementalHFiles:422 : Bulk load operation did not find
> any files to load in directory s3://privatekeybucket-
> anac5h41523l/kylin/kylin_default_instance/kylin-2c86b4b6-7639-4a97-ba63-
> 63c9dca095f6/kylin_sales_cube_clone3/hfile.  Does it contain files in
> subdirectories that correspond to column family names?
>
> In S3 view, I see the files exist in "_temporary" folder, seems were not
> moved to the target folder on complete. It seems EMR try to direct write to
> otuput path, but actually not.
>
> 2017-08-11 16:34 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>
>> No, defaultFs is hdfs.
>>
>> I’ve seen such behavior when set working dir to s3, but didn’t set
>> cluster-fs at all. Maybe you have a typo in the name of the property. I
>> used the old one «kylin.hbase.cluster.fs»
>>
>> When both working-dir and cluster-fs were set to s3 I got _temporary dir
>> of convert job at s3, but no hfiles. Also I saw correct output path for the
>> job in the log. But I didn’t check if job creates temporary files in s3,
>> but then copies results to hdfs. I hardly believe it happens.
>>
>> Do you see proper arguments for the step in the log?
>>
>>
>> 11 авг. 2017 г., в 11:17, ShaoFeng Shi <shaofeng...@apache.org>
>> написал(а):
>>
>> Hi Alexander,
>>
>> That makes sense. Using S3 for Cube build and storage is required for a
>> cloud hadoop environment.
>>
>> I tried to reproduce this problem. I created a EMR with S3 as HBase
>> storage, in kylin.properties, I set "kylin.env.hdfs-working-dir"
>> and "kylin.storage.hbase.cluster-fs" to the S3 bucket. But in the "Convert
>> Cuboid Data to HFile" step, Kylin still writes to local HDFS; Did you
>> modify the core-site.xml to make S3 as the default FS?
>>
>>
>>
>>
>> 2017-08-10 22:53 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>>
>>> Yes, I workarounded this problem in such way and it works.
>>>
>>> One problem of such solution is that I have to use pretty large hdfs and
>>> it'expensive. And also I have to manually garbage collect it, because it is
>>> not moved to s3, but copied. Kylin cleanup job doesn't work for it, because
>>> main metadata folder is at s3. So it would be really nice to put everything
>>> to s3.
>>>
>>> Another problem is that I had to rise hbase rpc timeout, because bulk
>>> loading from hdfs takes long. That was not trivial. 3 minutes work good,
>>> but with drawback of queries or metadata writes handing for 3 minutes if
>>> something bad happen. But that's rare event.
>>>
>>> 10 авг. 2017 г. 17:42 пользователь "ShaoFeng Shi" <
>>> shaofeng...@apache.org> написал:
>>>
>>> How about leaving empty for "kylin.hbase.cluster.fs"? This property is
>>>> for two-cluster deployment (one Hadoop for cube build, the other for
>>>> query);
>>>>
>>>> When be empty, the HFile will be written to default fs (HDFS in EMR),
>>>> and then load to HBase. I'm not sure whether EMR HBase (using S3 as
>>>> storage) can bulk load files from HDFS or not. If it can, that would be
>>>> great as the write performance of HDFS would be better than S3.
>>>>
>>>> 2017-08-10 22:29 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>>>>
>>>>> I also thought about it, but no, it's not consistency.
>>>>>
>>>>> Consistency view is enabled. I use same s3 for my own map-reduce jobs
>>>>> and it's ok.
>>>>>
>>>>> I also checked if it lost consistency (emrfs diff). No problems.
>>>>>
>>>>> In case of inconsistency of s3 files disappear right after they were
>>>>> written and appear some time after. Hfiles didn't appear after a day, but
>>>>> _template is there.
>>>>>
>>>>> It's 100% reproducable, I think I'll investigate this problem by
>>>>> running conversion job manually.
>>>>>
>>>>> 10 авг. 2017 г. 17:18 пользователь "ShaoFeng Shi" <
>>>>> shaofeng...@apache.org> написал:
>>>>>
>>>>> Did you enable the Consistent View? This article explains the
>>>>>> challenge when using S3 directly for ETL process:
>>>>>> https://aws.amazon.com/cn/blogs/big-data/ensuring-consistenc
>>>>>> y-when-using-amazon-s3-and-amazon-elastic-mapreduce-for-etl-
>>>>>> workflows/
>>>>>>
>>>>>>
>>>>>> 2017-08-09 18:19 GMT+08:00 Alexander Sterligov <sterligo...@joom.it>:
>>>>>>
>>>>>>> Yes, it's empty. Also I see this message in the log:
>>>>>>>
>>>>>>> 2017-08-09 09:02:35,947 WARN  [Job 
>>>>>>> 1e436685-7102-4621-a4cb-6472b866126d-7608]
>>>>>>> mapreduce.LoadIncrementalHFiles:234 : Skipping non-directory
>>>>>>> s3://joom.emr.fs/home/production/bi/kylin/kylin_metadata/kyl
>>>>>>> in-1e436685-7102-4621-a4cb-6472b866126d
>>>>>>> /main_event_1_main/hfile/_SUCCESS
>>>>>>> 2017-08-09 09:02:36,009 WARN  [Job 
>>>>>>> 1e436685-7102-4621-a4cb-6472b866126d-7608]
>>>>>>> mapreduce.LoadIncrementalHFiles:252 : Skipping non-file
>>>>>>> FileStatusExt{path=s3://joom.emr.fs/home/production/bi/kylin
>>>>>>> /kylin_metadata/kylin-1e436685-7102-4621-a4cb-6472b866126d/m
>>>>>>> ain_event_1_main/hfile/_temporary/1; isDirectory=true;
>>>>>>> modification_time=0; access_time=0; owner=; group=; 
>>>>>>> permission=rwxrwxrwx;
>>>>>>> isSymlink=false}
>>>>>>> 2017-08-09 09:02:36,014 WARN  [Job 
>>>>>>> 1e436685-7102-4621-a4cb-6472b866126d-7608]
>>>>>>> mapreduce.LoadIncrementalHFiles:422 : Bulk load operation did not
>>>>>>> find any files to load in directory s3://joom.emr.fs/home/producti
>>>>>>> on/bi/kylin/kylin_metadata/kylin-1e436685-7102-4621-a4cb-647
>>>>>>> 2b866126d/main_event_1_main/hfile.  Does it contain files in
>>>>>>> subdirectories that correspond to column family names?
>>>>>>>
>>>>>>> On Wed, Aug 9, 2017 at 1:15 PM, ShaoFeng Shi <shaofeng...@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> The HFile will be moved to HBase data folder when bulk load
>>>>>>>> finished; Did you check whether the HTable has data?
>>>>>>>>
>>>>>>>> 2017-08-09 17:54 GMT+08:00 Alexander Sterligov <sterligo...@joom.it
>>>>>>>> >:
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> I set kylin.hbase.cluster.fs to s3 bucket where hbase lives.
>>>>>>>>>
>>>>>>>>> Step "Convert Cuboid Data to HFile" finished without errors.
>>>>>>>>> Statistics at the end of the job said that it has written lot's of 
>>>>>>>>> data to
>>>>>>>>> s3.
>>>>>>>>>
>>>>>>>>> But there is no hfiles in kylin_metadata folder (kylin_metadata
>>>>>>>>> /kylin-1e436685-7102-4621-a4cb-6472b866126d/<table name>/hfile),
>>>>>>>>> but only _temporary folder and _SUCCESS file.
>>>>>>>>>
>>>>>>>>> _temporary contains hfiles inside attempt folders. it looks like
>>>>>>>>> there were not copied from _temporary to result dir. But there is no 
>>>>>>>>> errors
>>>>>>>>> neither in kylin log, nor in reducers' logs.
>>>>>>>>>
>>>>>>>>> Then loading empty hfiles produces empty segments.
>>>>>>>>>
>>>>>>>>> Is that a bug or I'm doing something wrong?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Shaofeng Shi 史少锋
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi 史少锋
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

Reply via email to