Re: Streaming Build Timestamp parse error

[Mario Copperfield]

I think it might make a confuse, because we usually define timestamp as an unix 
timestamp such as “1484236798000"

On 20 Jan 2017, 13:25 +0800, ShaoFeng Shi , wrote:
> "When i use timestamp as partition col, it occurs exception as follow:";
>
> Please aware that "1484236798000" isn't a timestamp value. Should use the 
> "MINUTE_START" or "HOUR_START" as the cube partition column, which are real 
> timestamp.
>
> > 2017-01-20 10:38 GMT+08:00 Mario Copperfield :
> > > Additionally, the stack log is in the following picture.
> > >
> > > On 20 Jan 2017, 10:27 +0800, Mario Copperfield , 
> > > wrote:
> > > > I’m sorry i don’t found the property “tsParser”, did you mean that 
> > > > “Parser Name”?
> > > >
> > > > The following picture is my configure.
> > > >
> > > > I change a little codes so that it can work but i found that the master 
> > > > branch has changed the class “UpdateCubeInfoAfterBuildStep” so much 
> > > > that I did not pull request.
> > > >
> > > > This is what my change:
> > > > https://github.com/xwhfcenter/kylin/commit/3e13bf244b82387107b5b445ae9946daf919cf54
> > > >
> > > >
> > > > On 19 Jan 2017, 22:31 +0800, ShaoFeng Shi , 
> > > > wrote:
> > > > > Kylin's DefaultTimeParser will parse the long typed timestamp value
> > > > > like "1484236798000",
> > > > > so in this case you don't need specify "tsParser".
> > > > >
> > > > > 2017-01-19 22:18 GMT+08:00 :
> > > > >
> > > > > > I'll check tomorrow
> > > > > >
> > > > > > Sent from Alto
> > > > > > On Thursday, January 19, 2017 at 22:10 ShaoFeng Shi <
> > > > > > shaofeng...@apache.org> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Could you provide the stack trace? Besdies, did you specifiy the 
> > > > > > "tsParser"
> > > > > > property when define the table? if yes, what's the value?
> > > > > >
> > > > > > 2017-01-19 21:45 GMT+08:00 Li Yang :
> > > > > >
> > > > > > > Deserves a JIRA I think.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Yang
> > > > > > >
> > > > > > > On Mon, Jan 16, 2017 at 2:34 PM, Copperfield wrote:
> > > > > > >
> > >
> > > > > > > > When i use timestamp as partition col, it occurs exception as 
> > > > > > > > follow:
> > > > > > > > java.text.ParseException: Unparseable date: "1484236798000" 
> > > > > > > > does not
> > > > > > > match
> > > > > > > > (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
> > > > > > > > \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)
> > > > > > > > my kylin version is 1.6.0
> > > > > > > >
> > > > > > > > --
> > > > > > > > View this message in context: http://apache-kylin.74782.x6.
> > > > > > > > nabble.com/Streaming-Build-Timestamp-parse-error-tp6952.html
> > > > > > > > Sent from the Apache Kylin mailing list archive at Nabble.com.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >
> > > > > > Shaofeng Shi 史少锋
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi 史少锋
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

org.apache.kylin.dict.CachedTreeMap use a couple classes from org.apache.hadoop.fs

[ShaoFeng Shi]

Hi Yerui,

I noticed that the CachedTreeMap.java uses a couple of classes
from org.apache.hadoop.fs package; and you have a comment "TODO Depends on
HDFS for now, ideally just depends on storage interface"

Now this impact the cube building with Spark, as some classes like
org.apache.hadoop.fs.Path isn't serializable while Spark relies on Java
serialization heavily. Will get error when building a cube with bitmap
measure as in below. So, can it be changed to ordinary classes like String
here? Thanks!

Caused by: java.io.NotSerializableException: org.apache.hadoop.fs.Path
Serialization stack:
- object not serializable (class: org.apache.hadoop.fs.Path, value:
hdfs:/kylin/kylin_default_instance/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP)
- writeObject data (class: java.util.TreeMap)
- object (class org.apache.kylin.dict.CachedTreeMap, {=null})
- field (class: org.apache.kylin.dict.AppendTrieDictionary, name:
dictSliceMap, type: class java.util.TreeMap)
- object (class org.apache.kylin.dict.AppendTrieDictionary,
AppendTrieDictionary(hdfs:///kylin/kylin_default_instance/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/))
- writeObject data (class: java.util.HashMap)
- object (class java.util.HashMap,
{DEFAULT.TEST_KYLIN_FACT.LSTG_SITE_ID=org.apache.kylin.dict.TrieDictionaryForest@f30773fa,
DEFAULT.TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME=org.apache.kylin.dict.TrieDictionaryForest@18259639,
DEFAULT.TEST_CATEGORY_GROUPINGS.META_CATEG_NAME=org.apache.kylin.dict.TrieDictionaryForest@44184626,
BUYER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_SELLER_LEVEL=org.apache.kylin.dict.TrieDictionaryForest@879f6439,
SELLER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_SELLER_LEVEL=org.apache.kylin.dict.TrieDictionaryForest@879f6439,
BUYER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_BUYER_LEVEL=org.apache.kylin.dict.TrieDictionaryForest@879f6439,
SELLER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_BUYER_LEVEL=org.apache.kylin.dict.TrieDictionaryForest@879f6439,
DEFAULT.TEST_KYLIN_FACT.TRANS_ID=org.apache.kylin.dict.TrieDictionaryForest@93b5aa11,
DEFAULT.TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME=org.apache.kylin.dict.TrieDictionaryForest@a494947b,
SELLER_COUNTRY:DEFAULT.TEST_COUNTRY.NAME
=org.apache.kylin.dict.TrieDictionaryForest@b3559b4c, BUYER_COUNTRY:
DEFAULT.TEST_COUNTRY.NAME
=org.apache.kylin.dict.TrieDictionaryForest@b3559b4c,
SELLER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_COUNTRY=org.apache.kylin.dict.TrieDictionaryForest@410216c0,
BUYER_ACCOUNT:DEFAULT.TEST_ACCOUNT.ACCOUNT_COUNTRY=org.apache.kylin.dict.TrieDictionaryForest@410216c0,
DEFAULT.TEST_KYLIN_FACT.PRICE=org.apache.kylin.dict.TrieDictionaryForest@89f144c6,
DEFAULT.TEST_KYLIN_FACT.TEST_COUNT_DISTINCT_BITMAP=AppendTrieDictionary(hdfs:///kylin/kylin_default_instance/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/),
DEFAULT.TEST_KYLIN_FACT.LEAF_CATEG_ID=org.apache.kylin.dict.TrieDictionaryForest@25e701d0,
DEFAULT.TEST_KYLIN_FACT.SLR_SEGMENT_CD=org.apache.kylin.dict.TrieDictionaryForest@dcfc7d11,
DEFAULT.TEST_KYLIN_FACT.CAL_DT=DateStrDictionary [pattern=-MM-dd,
baseId=0]})



-- 
Best regards,

Shaofeng Shi 史少锋

Re: Streaming Build Timestamp parse error

[xwhfcenter]

I'll check tomorrow

Sent from Alto
On Thursday, January 19, 2017 at 22:10 ShaoFeng Shi  
wrote:
Hi,

Could you provide the stack trace? Besdies, did you specifiy the "tsParser"
property when define the table? if yes, what's the value?

2017-01-19 21:45 GMT+08:00 Li Yang :

> Deserves a JIRA I think.
>
> Cheers
> Yang
>
> On Mon, Jan 16, 2017 at 2:34 PM, Copperfield wrote:
>
> > When i use timestamp as partition col, it occurs exception as follow:
> > java.text.ParseException: Unparseable date: "1484236798000" does not
> match
> > (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
> > \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)
> > my kylin version is 1.6.0
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Streaming-Build-Timestamp-parse-error-tp6952.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: Streaming Build Timestamp parse error

[ShaoFeng Shi]

Hi,

Could you provide the stack trace? Besdies, did you specifiy the "tsParser"
property when define the table? if yes, what's the value?

2017-01-19 21:45 GMT+08:00 Li Yang :

> Deserves a JIRA I think.
>
> Cheers
> Yang
>
> On Mon, Jan 16, 2017 at 2:34 PM, Copperfield  wrote:
>
> > When i use timestamp as partition col, it occurs exception as follow:
> > java.text.ParseException: Unparseable date: "1484236798000" does not
> match
> > (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
> > \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)
> > my kylin version is 1.6.0
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Streaming-Build-Timestamp-parse-error-tp6952.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: Streaming Build Timestamp parse error

[Li Yang]

Deserves a JIRA I think.

Cheers
Yang

On Mon, Jan 16, 2017 at 2:34 PM, Copperfield  wrote:

> When i use timestamp as partition col, it occurs exception as follow:
> java.text.ParseException: Unparseable date: "1484236798000" does not match
> (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
> \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)
> my kylin version is 1.6.0
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Streaming-Build-Timestamp-parse-error-tp6952.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>