Re: Only retrieving high cardinality dimension

[Kiriti Sai]

I don't want to group by or apply any filters for that dimension, I just
want to do all those query related things with other dimensions and just
get the corresponding description column value.
As I mentioned before, I'm not sure if such optimization exists in any OLAP
systems, but just wanted to clarify.

Thank you.
On Jan 8, 2016 7:19 PM, "hongbin ma"  wrote:

> how do you wish to use such dimension if we imaginarily had implemented it
> ?
> you can't apply groupby or filters on it, and it's obviously not a measure
> either.
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys (Lorg/apache/hadoop/conf/Configuration; ) Ljava/util/List;

[Kiriti Sai]

Hi,
I've recently update the binaries in my Kylin setup from v1.1 incubating to
v1.2. The cubes which were building fine till now are throwing the above
error.
This error is occuring in the extract fact table distinct columns step.
(Step 2).
Can you please point out any mistakes with the upgrading procedure or
anything else.

Thank you,
Sai Kiriti B.

Re: Java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys (Lorg/apache/hadoop/conf/Configuration; ) Ljava/util/List;

[Kiriti Sai]

Hi Wang,
The version of Hadoop in the cluster is 2.6. I've setup Hbase 0.98.15 using
binaries, in which there are some jar files of the Hadoop version 2.2, like
hadoop-yarn-client-2.2.jar.
As I've mentioned already, this setup has worked with the previous version
of Kylin 1.1-incubating, but has been throwing this error after updating to
v1.2. (Dont know if there is anything due to this, but just mentioning it).
So, is there any other to solve this other than building HBase from source
using the latest Hadoop libraries.

Thank You.
On Jan 5, 2016 8:26 PM, "Xiaoyu Wang" <wangxiao...@jd.com> wrote:

> Hi,
> The api YarnConfiguration.getServiceAddressConfKeys required Hadoop2.4+
> Which version Hadoop do you use ?
> You can recompile the hbase with hadoop 2.4+ version or your hadoop
> cluster version.
>
>
>
>
> 在 2016年01月05日 18:52, Kiriti Sai 写道:
>
>> Hi,
>> I have looked at the suggested link before posting the question here. I
>> didn't understand how to resolve this issue.
>> I've tried replacing the 2.2 hadoop yarn libs present in the HBase lib
>> directory but then it throws FileNotFoundException.
>> Can you please explain in a detailed way how to resolve this issue.
>> I'm using Hbase 0.98.15-hadoop2 version.
>>
>> Thank you,
>> Sai Kiriti B
>> On Jan 5, 2016 7:38 PM, "Xiaoyu Wang" <wangxiao...@jd.com> wrote:
>>
>> Hi Sai!
>>> You can see the same topic :
>>>
>>>
>>> http://apache-kylin.74782.x6.nabble.com/NoSuchMethodError-org-apache-hadoop-yarn-conf-YarnConfiguration-getServiceAddressConfKeys-td2937.html#a2943
>>>
>>> 在 2016年01月05日 18:27, Kiriti Sai 写道:
>>>
>>> Hi,
>>>> I've recently update the binaries in my Kylin setup from v1.1 incubating
>>>> to
>>>> v1.2. The cubes which were building fine till now are throwing the above
>>>> error.
>>>> This error is occuring in the extract fact table distinct columns step.
>>>> (Step 2).
>>>> Can you please point out any mistakes with the upgrading procedure or
>>>> anything else.
>>>>
>>>> Thank you,
>>>> Sai Kiriti B.
>>>>
>>>>
>>>>
>

Re: Java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys (Lorg/apache/hadoop/conf/Configuration; ) Ljava/util/List;

[Kiriti Sai]

Hi,
Can you please explain it in a slightly detailed manner. I understand that
the url you are referring to is the resource manager url, but it's
particular to a job right? How can something particular to a job be set as
a property for Kylin. I'm sorry if I'm mistaken.
Or are you intending that {job_id} will actually get the id of the MR job
running? Sorry for these naive questions.

Thank you.
On Jan 5, 2016 8:42 PM, "Xiaoyu Wang" <wangxiao...@jd.com> wrote:

> Hi,
> You can set the property in kylin.properties file
> kylin.job.yarn.app.rest.check.status.url=
> https://YOUR_RM_AND_PORT/ws/v1/cluster/apps/${job_id}?anonymous=true
>
> 在 2016年01月05日 19:38, Kiriti Sai 写道:
>
>> Hi Wang,
>> The version of Hadoop in the cluster is 2.6. I've setup Hbase 0.98.15
>> using
>> binaries, in which there are some jar files of the Hadoop version 2.2,
>> like
>> hadoop-yarn-client-2.2.jar.
>> As I've mentioned already, this setup has worked with the previous version
>> of Kylin 1.1-incubating, but has been throwing this error after updating
>> to
>> v1.2. (Dont know if there is anything due to this, but just mentioning
>> it).
>> So, is there any other to solve this other than building HBase from source
>> using the latest Hadoop libraries.
>>
>> Thank You.
>> On Jan 5, 2016 8:26 PM, "Xiaoyu Wang" <wangxiao...@jd.com> wrote:
>>
>> Hi,
>>> The api YarnConfiguration.getServiceAddressConfKeys required Hadoop2.4+
>>> Which version Hadoop do you use ?
>>> You can recompile the hbase with hadoop 2.4+ version or your hadoop
>>> cluster version.
>>>
>>>
>>>
>>>
>>> 在 2016年01月05日 18:52, Kiriti Sai 写道:
>>>
>>> Hi,
>>>> I have looked at the suggested link before posting the question here. I
>>>> didn't understand how to resolve this issue.
>>>> I've tried replacing the 2.2 hadoop yarn libs present in the HBase lib
>>>> directory but then it throws FileNotFoundException.
>>>> Can you please explain in a detailed way how to resolve this issue.
>>>> I'm using Hbase 0.98.15-hadoop2 version.
>>>>
>>>> Thank you,
>>>> Sai Kiriti B
>>>> On Jan 5, 2016 7:38 PM, "Xiaoyu Wang" <wangxiao...@jd.com> wrote:
>>>>
>>>> Hi Sai!
>>>>
>>>>> You can see the same topic :
>>>>>
>>>>>
>>>>>
>>>>> http://apache-kylin.74782.x6.nabble.com/NoSuchMethodError-org-apache-hadoop-yarn-conf-YarnConfiguration-getServiceAddressConfKeys-td2937.html#a2943
>>>>>
>>>>> 在 2016年01月05日 18:27, Kiriti Sai 写道:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> I've recently update the binaries in my Kylin setup from v1.1
>>>>>> incubating
>>>>>> to
>>>>>> v1.2. The cubes which were building fine till now are throwing the
>>>>>> above
>>>>>> error.
>>>>>> This error is occuring in the extract fact table distinct columns
>>>>>> step.
>>>>>> (Step 2).
>>>>>> Can you please point out any mistakes with the upgrading procedure or
>>>>>> anything else.
>>>>>>
>>>>>> Thank you,
>>>>>> Sai Kiriti B.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>

Re: org.apache.hadoop.hive.ql.metadata.HiveException

[Kiriti Sai]

Hi,
This error is caused because there is no Snappy compression codec available
in your setup and Kylin expects it by default.
As a work around, you can disable the use of snappy in the configuration
files of Kylin.
> Comment the compression.codec line in kylin.properties
> comment the properties in the kylin_job_conf.xml which are related to
compression. I guess there are around 4 properties to be commented.

This was the work around I used for a while but its recommended to use
compression techniques to minimize the memory shuffling between reducers.

Thank you.
Sai Kiriti B
On Jan 5, 2016 12:31 PM, "和风" <363938...@qq.com> wrote:

> hi:
>   execution "bulid" cube, jobs exception :
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException
>
>
> logs:
>
>
> OS command error exit with 2 -- hive  -e "USE default;
> DROP TABLE IF EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> (
> DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> STORED AS SEQUENCEFILE
> LOCATION
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
>
>
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET mapred.compress.map.output=true;
> SET
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compress=true;
> SET
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.max.split.locations=2000;
> SET dfs.replication=2;
> SET hive.merge.mapfiles=true;
> SET hive.merge.mapredfiles=true;
> SET hive.merge.size.per.task=268435456;
> SET hive.support.concurrency=false;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask = true;
> SET hive.auto.convert.join.noconditionaltask.size = 3;
> INSERT OVERWRITE TABLE
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> SELECT
> KYLIN_CAL_DT.AGE_FOR_QTR_ID
> ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> ,KYLIN_CAL_DT.YEAR_ID
> FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> '2016-01-12')
> ;
>
>
> "
>
>
> Logging initialized using configuration in
> jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> OK
> Time taken: 0.936 seconds
> OK
> Time taken: 0.112 seconds
> OK
> Time taken: 0.438 seconds
> Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1449731904014_1636, Tracking URL =
> http://cloud001:8088/proxy/application_1449731904014_1636/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill
> job_1449731904014_1636
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
> 2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_1449731904014_1636 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1449731904014_1636_m_00 (and more) from job
> job_1449731904014_1636
>
>
> Task with the most failures(1):
> -
> Task ID:
>   task_1449731904014_1636_m_00
>
>
> URL:
>
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636=task_1449731904014_1636_m_00
> -
> Diagnostic Messages for this Task:
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> 

Re: Select * not returning any rows

[Kiriti Sai]

Hi,
Thank you Hongbin Ma for suggestion to upgrade.
I've just now changed the version of Kylin to 1.2 and I'm able to see some
results when I perform 'select *' on the tables. But the number of results
returned are not exactly as expected.
In my case, I have 100 rows in a table, some of which might be repeated.
But when I'm using select *, it's kind of performing group by over all the
columns in the cube and returning very few rows.
Is this behavior intended or a bug?
If the second, please let me know if there is a way to correct it.

Thank you guys for responding quickly on new year's eve.
On Dec 31, 2015 4:04 PM, "Shi, Shaofeng" <shao...@ebay.com> wrote:

> This “how to_upgrade” is specifically for v0.6 to v0.7 upgrade, not for
> other versions;
>
> Between v0.7 to v1.2, the metadata is compatible; What user need do is
> just backup and restore the $KYLIN_HOME/conf folder after switch to a new
> Kylin binary; All the cube metadata and cube data are in HBase so no need
> to re-create or rebuild;
>
> On 12/31/15, 2:49 PM, "250635...@qq.com" <250635...@qq.com> wrote:
>
> >Have you checked out this one ?
> >http://kylin.apache.org/docs/howto/howto_upgrade.html
> >
> >
> >
> >
> >250635...@qq.com
> >
> >From: Kiriti Sai
> >Date: 2015-12-31 14:36
> >To: dev
> >Subject: Re: Select * not returning any rows
> >Since I'm just working with binaries, can you please explain how to
> >upgrade
> >from v1.1 to v1.2?
> >Should I just extract and replace the whole folder or should I backup and
> >restore the data also in some way?
> >
> >Thank you for the immediate response. :)
> >On Dec 31, 2015 3:25 PM, "hongbin ma" <mahong...@apache.org> wrote:
> >
> >> ​i believe this issue has been fixed in v1.2, why not use the latest
> >> version?
> >>
> >>
> >> --
> >> Regards,
> >>
> >> *Bin Mahone | 马洪宾*
> >> Apache Kylin: http://kylin.io
> >> Github: https://github.com/binmahone
> >>
>
>

Re: Select * not returning any rows

[Kiriti Sai]

Since I'm just working with binaries, can you please explain how to upgrade
from v1.1 to v1.2?
Should I just extract and replace the whole folder or should I backup and
restore the data also in some way?

Thank you for the immediate response. :)
On Dec 31, 2015 3:25 PM, "hongbin ma"  wrote:

> ​i believe this issue has been fixed in v1.2, why not use the latest
> version?
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Select * not returning any rows

[Kiriti Sai]

Hi,
I've been using Kylin v1.1 setup with binaries on an EC2 cluster of 8
nodes. I've completed the setup and able to build some cubes apart from the
sample cube.
But when I try to query the cubes using select * on any table, including
the sample tables, it doesn't return any rows. It just shows the column
names and the number of results is shown as 1.