Step13 create Htable
Kylin.log shows "Job Fetcher: 1 running, 1 actual running, 0 ready, 0 others"
all the time
[pool-7-thread-10]:[2015-12-30
09:58:56,406][INFO][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:58)]
- -cubename kylin_sales_cube -input
hdfs://bihbasemaster/kylin/kylin_metadata/kylin-e09d19dd-2882-4cd2-b80c-468bbc98e3c2/kylin_sales_cube/rowkey_stats/part-r-00000
-htablename KYLIN_Q9IBVXLXUX
[pool-7-thread-10]:[2015-12-30
09:58:57,532][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.run(CreateHTableJob.java:106)]
- hbase will use lzo to compress data
[pool-7-thread-10]:[2015-12-30
09:58:57,533][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.run(CreateHTableJob.java:106)]
- hbase will use lzo to compress data
[pool-7-thread-10]:[2015-12-30
09:58:57,560][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.getSplits(CreateHTableJob.java:188)]
- 1 regions
[pool-7-thread-10]:[2015-12-30
09:58:57,560][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.getSplits(CreateHTableJob.java:189)]
- 0 splits
[pool-7-thread-10]:[2015-12-30
09:58:57,622][INFO][org.apache.kylin.job.tools.DeployCoprocessorCLI.addCoprocessorOnHTable(DeployCoprocessorCLI.java:119)]
- Add coprocessor on KYLIN_Q9IBVXLXUX
[pool-7-thread-10]:[2015-12-30
09:58:57,623][INFO][org.apache.kylin.job.tools.DeployCoprocessorCLI.deployCoprocessor(DeployCoprocessorCLI.java:99)]
- hbase table [B@3c8745b2 deployed with coprocessor.
[pool-6-thread-1]:[2015-12-30
09:59:13,947][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
- Job Fetcher: 1 running, 1 actual running, 0 ready, 0 others
[pool-6-thread-1]:[2015-12-30
10:00:13,947][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
- Job Fetcher: 1 running, 1 actual running, 0 ready, 0 others
[pool-6-thread-1]:[2015-12-30
10:01:13,947][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
- Job Fetcher: 1 running, 1 actual running, 0 ready, 0 others
[pool-6-thread-1]:[2015-12-30
10:02:13,947][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
- Job Fetcher: 1 running, 1 actual running, 0 ready, 0 others
But the table has been created already
hbase(main):001:0> desc 'KYLIN_Q9IBVXLXUX'
Table KYLIN_Q9IBVXLXUX is ENABLED
KYLIN_Q9IBVXLXUX, {TABLE_ATTRIBUTES => {coprocessor$1 =>
'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.2-3.jar|org.apac
he.kylin.storage.hbase.coprocessor.endpoint.IIEndpoint|1000|', coprocessor$2 =>
'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coproce
ssor-1.2-3.jar|org.apache.kylin.storage.hbase.coprocessor.observer.AggregateRegionObserver|1001|',
METADATA => {'KYLIN_HOST' => 'kylin_metadata', 'S
PLIT_POLICY' =>
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}}
COLUMN FAMILIES DESCRIPTION
{NAME => 'F1', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'LZO', MIN_VERSIO
NS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE =>
'4194304', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
{NAME => 'F2', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'LZO', MIN_VERSIO
NS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE =>
'4194304', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
2 row(s) in 1.6060 seconds
hbase(main):002:0>
-----邮件原件-----
发件人: Luke Han [mailto:[email protected]]
发送时间: 2015年12月30日 8:30
收件人: [email protected]
主题: Re: long time create HTable without error
Hi Zhihua,
For more detail, please check installation guide first:
http://kylin.apache.org/docs/install/index.html
<http://kylin.apache.org/docs/install/index.html>
No matter which distribution you pickup, but please just try to play with
suggested one for quick start and then try to add your dataset.
Thanks.
Best Regards!
---------------------
Luke Han
On Tue, Dec 29, 2015 at 11:12 PM, Luke Han <[email protected]> wrote:
> I would like to suggest start experimentation with Hortonworks sandbox
> (HDP) and Kylin binary package first, when you have more insight about
> the entire process and how Kylin interactive with Hadoop/Hive/HBase,
> then move to your own cluster with apache only distribution.
>
> And please double check in your kylin.log file, there's should have
> more detail idea about what's wrong.
>
> Thanks.
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Tue, Dec 29, 2015 at 10:08 PM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
> [email protected]> wrote:
>
>> Hi all,
>>
>> Hadoop2.7.1
>> Hbase0.98.16
>> Hive1.2.1
>>
>> My sample cube Is running with error and error, this time
>> stoped at step 13 “Create HTable” for a long time without any error
>> notification
>>
>>
>> I found the log of hamster, as below
>>
>>
>> 2015-12-29 21:18:02,345 INFO [FifoRpcScheduler.handler1-thread-18]
>> master.HMaster: Client=root/null create 'KYLIN_VSPE13YZKF',
>> {TABLE_ATTRIBUTES => {coprocessor$1 =>
>> 'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coproces
>> sor-1.2-1.jar|org.apache.kylin.storage.hbase.coprocessor.endpoint.IIE
>> ndpoint|1000|',
>> coprocessor$2 =>
>> 'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coproces
>> sor-1.2-1.jar|org.apache.kylin.storage.hbase.coprocessor.observer.Agg
>> regateRegionObserver|1001|', METADATA => {'KYLIN_HOST' =>
>> 'kylin_metadata', 'SPLIT_POLICY' =>
>> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}
>> }, {NAME => 'F1', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER =>
>> 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION =>
>> 'LZO', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
>> 'FALSE', BLOCKSIZE => '4194304', IN_MEMORY => 'false', BLOCKCACHE =>
>> 'true'}, {NAME => 'F2', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>> BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
>> COMPRESSION => 'LZO', MIN_VERSIONS => '0', TTL => 'FOREVER',
>> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '4194304', IN_MEMORY =>
>> 'false', BLOCKCACHE => 'true'}
>> 2015-12-29 21:18:02,368 INFO
>> [MASTER_TABLE_OPERATIONS-dw-server20-30:60000-0]
>> handler.CreateTableHandler: Create table KYLIN_VSPE13YZKF
>> 2015-12-29 21:18:02,393 INFO
>> [RegionOpenAndInitThread-KYLIN_VSPE13YZKF-1] regionserver.HRegion:
>> creating HRegion KYLIN_VSPE13YZKF HTD == 'KYLIN_VSPE13YZKF',
>> {TABLE_ATTRIBUTES =>
>> {coprocessor$1 =>
>> 'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coproces
>> sor-1.2-1.jar|org.apache.kylin.storage.hbase.coprocessor.endpoint.IIE
>> ndpoint|1000|',
>> coprocessor$2 =>
>> 'hdfs://bihbasemaster/kylin/kylin_metadata/coprocessor/kylin-coproces
>> sor-1.2-1.jar|org.apache.kylin.storage.hbase.coprocessor.observer.Agg
>> regateRegionObserver|1001|', METADATA => {'KYLIN_HOST' =>
>> 'kylin_metadata', 'SPLIT_POLICY' =>
>> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}
>> }, {NAME => 'F1', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER =>
>> 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION =>
>> 'LZO', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
>> 'FALSE', BLOCKSIZE => '4194304', IN_MEMORY => 'false', BLOCKCACHE =>
>> 'true'}, {NAME => 'F2', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>> BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
>> COMPRESSION => 'LZO', MIN_VERSIONS => '0', TTL => 'FOREVER',
>> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '4194304', IN_MEMORY =>
>> 'false', BLOCKCACHE => 'true'} RootDir =
>> hdfs://bihbasemaster/hbase/.tmp Table name == KYLIN_VSPE13YZKF
>> 2015-12-29 21:18:02,410 INFO
>> [RegionOpenAndInitThread-KYLIN_VSPE13YZKF-1] regionserver.HRegion:
>> Closed KYLIN_VSPE13YZKF,,1451395082310.ce534b19cceff6d3c8f5866fbbf9fd47.
>> 2015-12-29 21:18:02,417 INFO
>> [MASTER_TABLE_OPERATIONS-dw-server20-30:60000-0] catalog.MetaEditor:
>> Added 1
>> 2015-12-29 21:18:02,418 INFO
>> [MASTER_TABLE_OPERATIONS-dw-server20-30:60000-0] master.AssignmentManager:
>> Bulk assigning 1 region(s) across 4 server(s), round-robin=true
>> 2015-12-29 21:18:02,424 INFO
>> [dw-server20-30.wlt,60000,1451383344930-GeneralBulkAssigner-3]
>> master.AssignmentManager: dw-server20-34.wlt,60020,1451383345797
>> unassigned
>> znodes=1 of total=1
>> 2015-12-29 21:18:02,424 INFO
>> [dw-server20-30.wlt,60000,1451383344930-GeneralBulkAssigner-3]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=OFFLINE, ts=1451395082418, server=null} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=PENDING_OPEN,
>> ts=1451395082424, server=dw-server20-34.wlt,60020,1451383345797}
>> 2015-12-29 21:18:02,435 INFO
>> [MASTER_TABLE_OPERATIONS-dw-server20-30:60000-0] master.AssignmentManager:
>> Bulk assigning done
>> 2015-12-29 21:18:02,438 INFO [AM.ZK.Worker-pool2-t127]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=PENDING_OPEN, ts=1451395082424,
>> server=dw-server20-34.wlt,60020,1451383345797} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=OPENING, ts=1451395082438,
>> server=dw-server20-34.wlt,60020,1451383345797}
>> 2015-12-29 21:18:02,444 INFO
>> [MASTER_TABLE_OPERATIONS-dw-server20-30:60000-0]
>> handler.CreateTableHandler: failed. Null
>> 2015-12-29 21:18:02,481 INFO [AM.ZK.Worker-pool2-t128]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=OPENING, ts=1451395082438,
>> server=dw-server20-34.wlt,60020,1451383345797} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=CLOSED, ts=1451395082481,
>> server=dw-server20-34.wlt,60020,1451383345797}
>> 2015-12-29 21:18:02,481 INFO [AM.ZK.Worker-pool2-t128]
>> master.AssignmentManager: Setting node as OFFLINED in ZooKeeper for
>> region {ENCODED => ce534b19cceff6d3c8f5866fbbf9fd47, NAME =>
>> 'KYLIN_VSPE13YZKF,,1451395082310.ce534b19cceff6d3c8f5866fbbf9fd47.',
>> STARTKEY => '', ENDKEY => ''}
>> 2015-12-29 21:18:02,481 INFO [AM.ZK.Worker-pool2-t128]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=CLOSED, ts=1451395082481,
>> server=dw-server20-34.wlt,60020,1451383345797} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=OFFLINE, ts=1451395082481,
>> server=dw-server20-34.wlt,60020,1451383345797}
>> 2015-12-29 21:18:02,484 INFO [AM.ZK.Worker-pool2-t128]
>> master.AssignmentManager: Assigning
>> KYLIN_VSPE13YZKF,,1451395082310.ce534b19cceff6d3c8f5866fbbf9fd47. to
>> dw-server20-31.wlt,60020,1451383345912
>> 2015-12-29 21:18:02,484 INFO [AM.ZK.Worker-pool2-t128]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=OFFLINE, ts=1451395082481,
>> server=dw-server20-34.wlt,60020,1451383345797} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=PENDING_OPEN,
>> ts=1451395082484, server=dw-server20-31.wlt,60020,1451383345912}
>> 2015-12-29 21:18:02,500 INFO [AM.ZK.Worker-pool2-t129]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=PENDING_OPEN, ts=1451395082484,
>> server=dw-server20-31.wlt,60020,1451383345912} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=OPENING, ts=1451395082500,
>> server=dw-server20-31.wlt,60020,1451383345912}
>> 2015-12-29 21:18:02,546 INFO [AM.ZK.Worker-pool2-t131]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=OPENING, ts=1451395082501,
>> server=dw-server20-31.wlt,60020,1451383345912} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=CLOSED, ts=1451395082546,
>> server=dw-server20-31.wlt,60020,1451383345912}
>> 2015-12-29 21:18:02,546 INFO [AM.ZK.Worker-pool2-t131]
>> master.AssignmentManager: Setting node as OFFLINED in ZooKeeper for
>> region {ENCODED => ce534b19cceff6d3c8f5866fbbf9fd47, NAME =>
>> 'KYLIN_VSPE13YZKF,,1451395082310.ce534b19cceff6d3c8f5866fbbf9fd47.',
>> STARTKEY => '', ENDKEY => ''}
>> 2015-12-29 21:18:02,546 INFO [AM.ZK.Worker-pool2-t131]
>> master.RegionStates: Transition {ce534b19cceff6d3c8f5866fbbf9fd47
>> state=CLOSED, ts=1451395082546,
>> server=dw-server20-31.wlt,60020,1451383345912} to
>> {ce534b19cceff6d3c8f5866fbbf9fd47 state=OFFLINE, ts=1451395082546,
>> server=dw-server20-31.wlt,60020,1451383345912}
>>
>>
>>
>>
>>
>> *********************************************************************
>> ***********************************************************
>> The information in this email is confidential and may be legally
>> privileged. If you have received this email in error or are not the
>> intended recipient, please immediately notify the sender and delete
>> this message from your computer. Any use, distribution, or copying of
>> this email other than by the intended recipient is strictly
>> prohibited. All messages sent to and from us may be monitored to
>> ensure compliance with internal policies and to protect our business.
>> Emails are not secure and cannot be guaranteed to be error free as
>> they can be intercepted, amended, lost or destroyed, or contain
>> viruses. Anyone who communicates with us by email is taken to accept these
>> risks.
>>
>> 收发邮件者请注意：
>> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
>> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
>>
>> *********************************************************************
>> ***********************************************************
>>
>
>
********************************************************************************************************************************
The information in this email is confidential and may be legally privileged. If
you have received this email in error or are not the intended recipient, please
immediately notify the sender and delete this message from your computer. Any
use, distribution, or copying of this email other than by the intended
recipient is strictly prohibited. All messages sent to and from us may be
monitored to ensure compliance with internal policies and to protect our
business.
Emails are not secure and cannot be guaranteed to be error free as they can be
intercepted, amended, lost or destroyed, or contain viruses. Anyone who
communicates with us by email is taken to accept these risks.
收发邮件者请注意：
本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
********************************************************************************************************************************
