regionserver logs on your server would tell you what happened during the ingestion.
BTW Mob feature is not in 1.1.x releases. You're likely using a vendor's backport. There have been continuous improvements to Mob feature since its initial checkin. So there may be some difference in the details between the release you use and hbase 2.0 (which I used to generate the logs I quoted). On Sat, May 19, 2018 at 11:29 AM, Nicolas Paris <nipari...@gmail.com> wrote: > 2018-05-19 20:08 GMT+02:00 Ted Yu <yuzhih...@gmail.com>: > > > Mob store file is renamed from /apps/hbase/data/mobdir to the final > > location under region directory for the table. > > > > This explains why you don't see much data under mobdir since data > ingestion > > has finished. > > > > > Well, I monitored the mobdir folder during ingestion. Nothing happened in > it. > Data were directly going under the table in the region. > > BTW, thats why I was thinking the pdf were treated as regular binary files. > Certainly reading the regionserver logs will help. > > Finallly, if hive is able to load mob columns, that's a good news for me. > > > > > > > > Cheers > > > > On Sat, May 19, 2018 at 9:58 AM, Nicolas Paris <nipari...@gmail.com> > > wrote: > > > > > Not having access cluster for few days, but I will be looking > > > to logs. > > > > > > However, when looking at your logs, it seems that I mispell > > > my mlob dir in the first post. It was "mobdir". > > > The /apps/hbase/data/mobdir/ is nearly empty, sizing 4 or 10 kb > > > > > > Would this confirm the mob flushing process wouldn't be activated ? > > > > > > > > > > > > 2018-05-19 18:38 GMT+02:00 Ted Yu <yuzhih...@gmail.com>: > > > > > > > If you have a chance to look at region server log, you would see some > > > line > > > > such as the following: > > > > > > > > 2018-05-19 16:31:23,548 INFO [MemStoreFlusher.0] > > regionserver.HMobStore: > > > > Renaming flushed file from > > > > hdfs://mycluster/apps/hbase/data/mobdir/.tmp/ > > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb > > 06fb9e800377 > > > > to > > > > hdfs://mycluster/apps/hbase/data/mobdir/data/default/ > > > > IntegrationTestIngestWithMOB/e9b5d936e7f55a4f1c3246a8d5ce53 > c2/test_cf/ > > > > 28e252d7f013973174750d483d358fa020180519dd8e7c3d67814eb0b5fb > > 06fb9e800377 > > > > > > > > Meaning Mob store file is first saved under > > > /apps/hbase/data/mobdir/.tmp/ , > > > > then renamed to under the usual location under region directory for > the > > > > table. > > > > > > > > From high level, as long as you can query what you ingested, you can > be > > > > assured that Mob data is persisted. > > > > > > > > Cheers > > > > > > > > On Sat, May 19, 2018 at 8:43 AM, Nicolas Paris <nipari...@gmail.com> > > > > wrote: > > > > > > > > > Hi > > > > > > > > > > Yes the data comes back as expected. > > > > > My table is not called "mlob" however since I found such folder > > > > > I thought it was storing mob objects. > > > > > > > > > > I do have 500 folder hashed as you mentionned. They contains the > > > > > whole dataset (2TO) > > > > > However, how beeing sure the data is actually stored as MOB (and > not > > > > > as traditional binary) > > > > > > > > > > Thanks > > > > > > > > > > > > > > > 2018-05-19 15:59 GMT+02:00 Ted Yu <yuzhih...@gmail.com>: > > > > > > > > > > > bq. look into hdfs hbase/data/mlob > > > > > > > > > > > > Is 'mlob' name of your table ? > > > > > > > > > > > > bq. nearly empty folder > > > > > > > > > > > > Here is listing under a one region table: > > > > > > > > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > > > > /apps/hbase/data/data/default/atlas_janus/.tabledesc > > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-16 23:51 > > > > > > /apps/hbase/data/data/default/atlas_janus/.tmp > > > > > > drwxr-xr-x - hbase hdfs 0 2018-05-17 00:55 > > > > > > /apps/hbase/data/data/default/atlas_janus/ > > > > 8033ea259cb7272d43bc137ca0ab29 > > > > > 06 > > > > > > > > > > > > Not sure if the above matches your description of being nearly > > empty. > > > > > > Here data is stored under 8033ea259cb7272d43bc137ca0ab2906 > > > > > > > > > > > > If you query the table, does the data come back as expected ? > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Sat, May 19, 2018 at 12:23 AM, Nicolas Paris < > > nipari...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > > > > > > > > I am using hbase 1.1 and hive 1.2 > > > > > > > > > > > > > > I created an hbase table with a mob column with the default > > > > > > > threshold (100K) > > > > > > > I mapped the table into hive with a binary format, and loaded > > > > > > > 20M of pdf of size between 50k and 20mb > > > > > > > > > > > > > > Apparently the mob is not populated because when I look into > > > > > > > hdfs hbase/data/mlob, it is a nearly empty folder. > > > > > > > > > > > > > > Does it mean hive cannot populate hbase mob columns ? > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > >
