sorry, for the java remote debug stuff, please also add "jpda" before the
"start" parameter in the tail part below
hbase ${JAVA_OPTS} $JPDA_OPTS \
-Djava.util.logging.config.file=${tomcat_root}/conf/logging.properties \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true \
-Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true \
-Djava.endorsed.dirs=${tomcat_root}/endorsed \
-Dcatalina.base=${tomcat_root} \
-Dcatalina.home=${tomcat_root} \
-Djava.io.tmpdir=${tomcat_root}/temp \
-Dkylin.hive.dependency=${hive_dependency} \
-Dspring.profiles.active=${spring_profile} \
org.apache.hadoop.util.RunJar ${tomcat_root}/bin/bootstrap.jar
org.apache.catalina.startup.Bootstrap jpda start
2015-03-16 18:06 GMT+08:00 Li Yang <[email protected]>:
> Hi Dong, could you give some numbers on |A|, |B|, and the latency of the
> two queries?
>
> Observation is usually the performance is IO bound. However for the case
> you mentioned, the queries have the same scan range, which means the same
> level of IO. So they SHOULD return in similar latency. Could it be hot/cold
> cache?
>
> The evaluation of IN is done by a set, so the size shall not matter.
>
> On Mon, Mar 16, 2015 at 5:11 PM, dong wang <[email protected]> wrote:
>
> > hi shaofeng, one more question, suppose that we have SQL like: select *
> > from t1 where col1 in {A} group by..
> >
> > set {A} is a subset of set {C}, say {B} = {C} - {A}, then {B} is
> > the difference set, and suppose that |A| > |B|, |A| indicates
> > A's cardinality, then tests show the performance of select * from t1
> where
> > col1 NOT IN {B} group by.. is much better that the performance of select
> *
> > from t1 where col1 IN {A} group by.. especially when |A| is much
> greater
> > than |B|, so, could anyone please have a explain?
> >
> > |
> >
> > 2015-03-16 16:11 GMT+08:00 Shi, Shaofeng <[email protected]>:
> >
> > > Awesome! Thanks for the sharing, we will compose these tips into wiki
> for
> > > all Kylin users;
> > >
> > > On 3/16/15, 4:03 PM, "dong wang" <[email protected]> wrote:
> > >
> > > >it works, thanks~
> > > >
> > > >adding more, if someone deploys kylin on linux server but wants to
> debug
> > > >kylin from Windows' source code in eclipse, one can just add the
> > following
> > > >codes in kylin.sh, and then start eclipse java remote debug feature~
> > > >
> > > >if [ -z "$JPDA_TRANSPORT" ]; then
> > > > JPDA_TRANSPORT="dt_socket"
> > > >fi
> > > >if [ -z "$JPDA_ADDRESS" ]; then
> > > > JPDA_ADDRESS="8084" #remote debug port
> > > >fi
> > > >if [ -z "$JPDA_SUSPEND" ]; then
> > > > JPDA_SUSPEND="n"
> > > >fi
> > > >if [ -z "$JPDA_OPTS" ]; then
> > > >
> > >
> >
> >JPDA_OPTS="-agentlib:jdwp=transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,
> > > >server=y,suspend=$JPDA_SUSPEND"
> > > >fi
> > > >
> > > >export JAVA_OPTS="-Xms2048M -Xmx2048M -Xss4M"
> > > >
> > > >
> > > >hbase ${JAVA_OPTS} $JPDA_OPTS \
> > >
> >-Djava.util.logging.config.file=${tomcat_root}/conf/logging.properties \
> > > >
> > > >2015-03-16 15:43 GMT+08:00 Shi, Shaofeng <[email protected]>:
> > > >
> > > >> Seems the JVM default stack size couldn¹t fulfill your need; You can
> > try
> > > >> to give a bigger stack size when start Kylin:
> > > >>
> > > >> in kylin.sh:
> > > >>
> > > >> export JAVA_OPTS="-Xms2048M -Xmx2048M -Xss4M"
> > > >>
> > > >> hbase ${JAVA_OPTS}
> > > >>
> > -Djava.util.logging.config.file=${CATALINA_HOME}/conf/logging.properties
> > > >> -Djava.library.path=${KYLIN_LD_LIBRARY_PATH}
> > > >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > > >> -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
> > > >> -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true
> > > >> -Dspring.profiles.active=sandbox
> > > >> -Djava.endorsed.dirs=${CATALINA_HOME}/endorsed
> > > >> -Dcatalina.base=${CATALINA_HOME} -Dcatalina.home=${CATALINA_HOME}
> > > >> -Djava.io.tmpdir=${CATALINA_HOME}/temp
> org.apache.hadoop.util.RunJar
> > > >> ${CATALINA_HOME}/bin/bootstrap.jar
> > > >>org.apache.catalina.startup.Bootstrap
> > > >> start > ${CATALINA_HOME}/logs/kylin.log 2>&1 &
> > > >>
> > > >>
> > > >> Please let us know whether it can solve the problem, so that it can
> > help
> > > >> other guys;
> > > >>
> > > >> On 3/16/15, 3:25 PM, "dong wang" <[email protected]> wrote:
> > > >>
> > > >> >when there are about 3500 values in the "select ...from.. where
> > > >>..in(v1,
> > > >> >v2, ...v3500)" syntax, it throw the error below, I'm debugging it
> > now,
> > > >> >since the problem is urgent, thus, hope that someone can also do a
> > > >>help
> > > >> >to
> > > >> >take a look?
> > > >> >
> > > >> >in addition, if the only a few values(for example, 10 values, and
> > > >>etc), it
> > > >> >can return correct result!
> > > >> >
> > > >> >[[http-bio-8080-exec-7]:[2015-03-16
> > > >>
> > >
> >
> >>>15:17:19,301][INFO][org.apache.kylin.query.routing.RoutingRule.applyRule
> > > >>>s(
> > > >> >RoutingRule.java:60)]
> > > >> >- ===================================================
> > > >> >[http-bio-8080-exec-7]:[2015-03-16
> > > >>
> > >
> >
> >>>15:17:19,301][INFO][org.apache.kylin.query.routing.QueryRouter.selectRea
> > > >>>li
> > > >> >zation(QueryRouter.java:54)]
> > > >> >- The realizations remaining:
> > > >> >[http-bio-8080-exec-7]:[2015-03-16
> > > >>
> > >
> >
> >>>15:17:19,301][INFO][org.apache.kylin.query.routing.QueryRouter.selectRea
> > > >>>li
> > > >> >zation(QueryRouter.java:55)]
> > > >> >- [table1]
> > > >> >[http-bio-8080-exec-7]:[2015-03-16
> > > >>
> > >
> >
> >>>15:17:19,301][INFO][org.apache.kylin.query.routing.QueryRouter.selectRea
> > > >>>li
> > > >> >zation(QueryRouter.java:56)]
> > > >> >- The realization being chosen: table1
> > > >> >[http-bio-8080-exec-7]:[2015-03-16
> > > >>
> > > >>>15:17:22,589][ERROR][
> > > org.apache.kylin.rest.controller.QueryController.do
> > > >>>Qu
> > > >> >ery(QueryController.java:226)]
> > > >> >- Exception when execute sql
> > > >> >java.lang.StackOverflowError
> > > >> > at
> > > >>
> > >
> >
> >>>org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:445
> > > >>>7)
> > > >> > at
> > > >>org.codehaus.janino.UnitCompiler.access$8900(UnitCompiler.java:183)
> > > >> > at
> > > >>
> > >
> >
> >>>org.codehaus.janino.UnitCompiler$11.visitBinaryOperation(UnitCompiler.ja
> > > >>>va
> > > >> >:4371)
> > > >> > at
> > org.codehaus.janino.Java$BinaryOperation.accept(Java.java:3768)
> > > >>
> > > >>
> > >
> > >
> >
>
