kylin developers
We have used superset as BI tool. Superset uses to_date to represent
time, and we add the to_date udf in our env. A query may be like select ***
from table_1 where 'DATE' >= TO_DATE('2017-12-31 00:00:00', 'yyyy-MM-dd').
The executing result is right, but the query will not use kylin optimize,
because some error has happened , and every query will scan all hbase
table. How should we add the udf ?
2018-01-30 13:10:03,400 WARN [Query
>> f029cbac-2aba-456c-b857-f65c8661e39c-90]
>> filter.BuiltInFunctionTupleFilter:143 : Reflection failed for methodParams.
>
> java.lang.NullPointerException
>
> at
>> org.apache.kylin.metadata.filter.BuiltInFunctionTupleFilter.addChild(BuiltInFunctionTupleFilter.java:136)
>
> at
>> org.apache.kylin.metadata.filter.TupleFilterSerializer.deserialize(TupleFilterSerializer.java:146)
>
> at
>> org.apache.kylin.storage.gtrecord.CubeSegmentScanner.<init>(CubeSegmentScanner.java:65)
>
> at
>> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.search(GTCubeStorageQueryBase.java:93)
>
> at
>> org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:117)
>
> at
>> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:64)
>
> at Baz$1$1.moveNext(Unknown Source)
>
> at
>> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
>
> at
>> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
>
> at
>> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
>
> at Baz.bind(Unknown Source)
>
> at
>> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:335)
>
> at
>> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
>
> at
>> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
>
> at
>> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
>
> at
>> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:204)
>
> at
>> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
>
> at
>> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
>
> at
>> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
>
> at
>> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:607)
>
> at
>> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
>
> at
>> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
>
> at
>> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
>
> at
>> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:845)
>
> at
>> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:572)
>
> at
>> org.apache.kylin.rest.service.QueryService.query(QueryService.java:181)
>
> at
>> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:428)
>
>
ToDateUDF likes this, and I have adding it on KylinConfigBase.getUDFs
public class ToDateUDF {
> private static final Logger logger =
> LoggerFactory.getLogger(ToDateUDF.class);
> public String eval(String sourceDateStr, String sourceDateFormat) {
> sourceDateStr = sourceDateStr.replaceAll("'", "").trim();
> sourceDateFormat = sourceDateFormat.replaceAll("'", "").trim();
> try {
> SimpleDateFormat dateFormat = new
> SimpleDateFormat(sourceDateFormat);
> long ts = dateFormat.parse(sourceDateStr).getTime();
> return getFormatTime(ts, "yyyyMMdd");
> } catch (ParseException e) {
> logger.error("parse error", e);
> logger.error("sourceDateStr:{},sourceDateFormat:{}",
> sourceDateStr, sourceDateFormat);
> return "";
> }
> }
> public static String getFormatTime(long timeStamp, String format) {
> if (StringUtils.isBlank(format)) {
> format = "yyyyMMdd";
> }
> Calendar cal = Calendar.getInstance();
> SimpleDateFormat formatter = new SimpleDateFormat(format);
> cal.setTimeInMillis(timeStamp);
> return formatter.format(cal.getTime());
> }
> }
