[jira] [Created] (KYLIN-3991) Provide more config options for Flink cube engine

2019-04-28 Thread vinoyang (JIRA) 
vinoyang created KYLIN-3991:
---

 Summary: Provide more config options for Flink cube engine
 Key: KYLIN-3991
 URL: https://issues.apache.org/jira/browse/KYLIN-3991
 Project: Kylin
  Issue Type: Sub-task
  Components: Flink Engine
Reporter: vinoyang
Assignee: vinoyang


Currently, there are some key configuration options has not been exposed to 
Kylin Flink engine, e.g. {{-yn}}. So we should expose more config options.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Why does the log always say “No Data Available” when the cube is built?

2019-04-28 Thread yuzhang 
Hi shiqi:
"No Data available" mean the step of the job hasn't been completed. There 
will be some log message if the step has been completed, whether successful or 
not.


For you problem, could you provide more detail about you build job? Such 
as, log on server, which step is running, your deploy environment, etc will be 
helpful.




Best regards


yuzhang


| |
yuzhang
|
|
shifengdefan...@163.com
|
签名由网易邮箱大师定制
On 4/28/2019 21:05，shiqi wrote：
In the sample case on the Kylin official website, when I was building cube,
in the first step of the Create Intermediate Flat Hive Table, the log is
always No Data Available, the status is always running.

The cube build has been executed for more than three hours.

I checked the hive database table kylin_sales and there is data in the
table.

And I fount that the intermediate flat hive table
kylin_intermediate_kylin_sales_cube_402e3eaa_dfb2_7e3e_04f3_07248c04c10c
has been created successfully in the hive, but there is no data in its.

```
hive> show tables;
OK
...
kylin_intermediate_kylin_sales_cube_402e3eaa_dfb2_7e3e_04f3_07248c04c10c
kylin_sales
...
Time taken: 9.816 seconds, Fetched: 1 row(s)

hive> select * from kylin_sales;
OK
...
89922012-04-17  ABIN15687   0   13  95.5336 17  19751507   
ADMIN   Shanghai
89932013-02-02  FP-non GTC  67698   0   13  85.7528 6   1856   
10004882MODELER Hongkong
...
Time taken: 3.759 seconds, Fetched: 1 row(s)
```

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Why does the log always say “No Data Available” when the cube is built?

2019-04-28 Thread shiqi 
In the sample case on the Kylin official website, when I was building cube,
in the first step of the Create Intermediate Flat Hive Table, the log is
always No Data Available, the status is always running.

The cube build has been executed for more than three hours.

I checked the hive database table kylin_sales and there is data in the
table.

And I fount that the intermediate flat hive table
kylin_intermediate_kylin_sales_cube_402e3eaa_dfb2_7e3e_04f3_07248c04c10c
has been created successfully in the hive, but there is no data in its.

```
hive> show tables;
OK
...
kylin_intermediate_kylin_sales_cube_402e3eaa_dfb2_7e3e_04f3_07248c04c10c
kylin_sales
...
Time taken: 9.816 seconds, Fetched: 1 row(s)

hive> select * from kylin_sales;
OK
...
89922012-04-17  ABIN15687   0   13  95.5336 17  19751507   
ADMIN   Shanghai
89932013-02-02  FP-non GTC  67698   0   13  85.7528 6   1856   
10004882MODELER Hongkong
...
Time taken: 3.759 seconds, Fetched: 1 row(s)
```

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

[jira] [Created] (KYLIN-3990) java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text

2019-04-28 Thread Colin Luo (JIRA) 
Colin Luo created KYLIN-3990:


 Summary: java.lang.ClassCastException: 
org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
 Key: KYLIN-3990
 URL: https://issues.apache.org/jira/browse/KYLIN-3990
 Project: Kylin
  Issue Type: Bug
Reporter: Colin Luo


HDP 3.1 Kylin 2.6.1

Error in build cube step 4

2019-04-28 19:32:31,762 INFO [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] orc.OrcInputFormat:1737 : ORC 
pushdown predicate: null
2019-04-28 19:32:32,139 INFO [ORC_GET_SPLITS #1] impl.OrcCodecPool:56 : Got 
brand-new codec ZLIB
2019-04-28 19:32:32,261 INFO [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] orc.OrcInputFormat:1855 : 
FooterCacheHitRatio: 0/1
2019-04-28 19:32:32,321 INFO [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] orc.ReaderImpl:82 : Reading ORC rows 
from 
hdfs://master1:8020/warehouse/tablespace/managed/hive/wmdp_etl_dm.db/dm_customer_type/base_011/bucket_0
 with \{include: null, offset: 3, length: 725, schema: 
struct,
 includeAcidColumns: true}
2019-04-28 19:32:32,391 INFO [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] mapreduce.InternalUtil:156 : 
Initializing org.apache.hadoop.hive.ql.io.orc.OrcSerde with properties 
\{columns.types=string,string,string,string, comment=客户类型维度, 
columns=customer_type_cd,customer_type_desc,customer_cate_cd,customer_cate_desc,
 transactional_properties=default, 
COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"customer_cate_cd":"true","customer_cate_desc":"true","customer_type_cd":"true","customer_type_desc":"true"}},
 serialization.format=1, numRows=11, numFiles=1, 
transient_lastDdlTime=1556442134, rawDataSize=0, columns.comments=客户类型代码 客户类型描述 
客户大类代码 客户大类描述, totalSize=1306, 
serialization.lib=org.apache.hadoop.hive.ql.io.orc.OrcSerde, 
serialization.null.format=\N, bucketing_version=2, transactional=true, 
column.name.delimiter=,, name=WMDP_ETL_DM.DM_CUSTOMER_TYPE}
2019-04-28 19:32:32,402 ERROR [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] common.HadoopShellExecutable:65 : 
error execute 
HadoopShellExecutable\{id=16f753e8-5444-1f03-178a-5cca4f3cd10f-03, name=Build 
Dimension Dictionary, state=RUNNING}
java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast 
to org.apache.hadoop.io.Text
 at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:46)
 at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:26)
 at 
org.apache.hive.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:278)
 at 
org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:199)
 at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
 at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
 at 
org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
 at 
org.apache.hive.hcatalog.data.transfer.impl.HCatInputFormatReader$HCatRecordItr.hasNext(HCatInputFormatReader.java:107)
 at org.apache.kylin.source.hive.HiveTableReader.next(HiveTableReader.java:90)
 at 
org.apache.kylin.dict.lookup.SnapshotTable.takeSnapshot(SnapshotTable.java:98)
 at 
org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:133)
 at 
org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1174)
 at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:1090)
 at 
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:97)
 at 
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
 at 
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
 at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
 at 
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
 at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
 at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
2019-04-28 19:32:32,407 INFO [Scheduler 2100016947 Job 
16f753e8-5444-1f03-178a-5cca4f3cd10f-202] execution.ExecutableManager:453 : job

Re: [New blog] "Real-time Streaming Design in Apache Kylin"

2019-04-28 Thread Li Yang 
Love to see this new direction!

On Mon, Apr 22, 2019 at 3:30 PM Iñigo Martínez 
wrote:

> Thank you, ShaoFeng.
>
> Very interesing. It's a more polished version of document attached at Jira
> feature request. ;)
>
> El jue., 18 abr. 2019 a las 4:28, ShaoFeng Shi ()
> escribió:
>
>> Hello,
>>
>> Gang Ma, the core developer of Kylin Real-time OLAP, just composed a tech
>> blog on this feature. It will help to understand the purpose, the
>> architecture and the design. Welcome to read and share with others:
>>
>> https://kylin.apache.org/blog/2019/04/12/rt-streaming-design/
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofeng...@apache.org
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscr...@kylin.apache.org
>> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>>
>>
>>
>
> --
>
>
>
>
> Iñigo Martínez
> Systems Manager
> imarti...@telecoming.com
>
>
>
>
>
>
>   [image: Telecoming - Make it digital]
> [image: 5000_empresas]
> [image: 1000_empresas]
>
> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
> telecoming.com 
>
>
>
>   Este correo electrónico y sus archivos adjuntos están dirigidos
> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
> confidencial, personal e intransferible del mismo está protegido
> legalmente. Cualquier publicación, reproducción, distribución o
> retransmisión no autorizada, ya sea completa o en parte, se encuentra
> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
> inmediatamente a la persona que lo ha enviado y borre el mensaje original
> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>
>

Re: [ANNOUNCE] Kaisen Kang joins the Apache Kylin PMC

2019-04-28 Thread Li Yang 
Welcome, Kaisen!

On Tue, Apr 16, 2019 at 2:20 PM JiaTao Tao  wrote:

> Congratulations!
>
> --
>
>
> Regards!
>
> Aron Tao
>
> Luke Han  于2019年4月16日周二 上午5:09写道：
>
> > On behalf of the Apache Kylin PMC I am pleased to announce that Kaisen
> Kang
> > has accepted our invitation to become a PMC member on the Apache Kylin
> > project. We appreciate Kaisen stepping up to take more responsibility in
> > the Kylin project.
> >
> > Please join me in welcoming Kaisen to the Kylin PMC!
> >
> > Best Regards,
> >
> > Luke
> >
>