Hi, I'm sorry for many confusing changes. I leave inline comments on your questions.
Best regards, Hyunsik Choi On Fri, Aug 30, 2013 at 5:02 PM, camelia c <[email protected]> wrote: > > Hello, > > I re-adapted a large part of my previous work to suit the new major changes > in Tajo. I documented the changes on the project's website and on Github. I > estimate that in a couple of days I will be able to move forward to the > final part of the project, i.e. to the physical operators. > > Now, I wanted to ask You for some advice about: > > PART A. QUESTIONS ABOUT RUNNING THE NEW TAJO VERSION > > 1) > I > have a problem because TAJO is not executing queries, not even for > queries without outer join. The log is the following. It seems that > there is a problem with the resource allocation > > 2013-08-28 > 11:44:39,965 INFO event.AsyncDispatcher > (AsyncDispatcher.java:register(153)) - Registering class > org.apache.tajo.master.querymaster.QueryJobEvent$Type for class > org.apache.tajo.master.querymaster.QueryInProgress$QueryInProgressEventHandler > 2013-08-28 11:44:39,965 INFO service.AbstractService > (AbstractService.java:init(81)) - Service:Dispatcher is inited. > 2013-08-28 11:44:39,965 INFO > service.AbstractService (AbstractService.java:init(81)) - > Service:org.apache.tajo.master.querymaster.QueryInProgress is inited. > 2013-08-28 11:44:39,966 INFO service.AbstractService > (AbstractService.java:start(94)) - Service:Dispatcher is started. > 2013-08-28 > 11:44:39,966 INFO service.AbstractService > (AbstractService.java:start(94)) - > Service:org.apache.tajo.master.querymaster.QueryInProgress is started. > 2013-08-28 > 11:44:39,966 INFO querymaster.QueryInProgress > (QueryInProgress.java:startQueryMaster(140)) - Initializing > QueryInProgress for QueryID=q_1377678790404_0001 > 2013-08-28 > 11:44:39,966 WARN rm.TajoWorkerResourceManager > (TajoWorkerResourceManager.java:allocateQueryMaster(101)) - No available > resource for querymaster:q_1377678790404_0001 > 2013-08-28 > 11:44:39,983 INFO rm.TajoWorkerResourceManager > (TajoWorkerResourceManager.java:run(196)) - ====> > allocateWorkerResources:eb_0000000000000_0000_000000, > required:1, allocated:0, queryMasterRequest=true, liveWorkers=0 > 2013-08-28 > 11:44:40,460 INFO rm.TajoWorkerResourceManager > (TajoWorkerResourceManager.java:run(196)) - ====> > allocateWorkerResources:eb_0000000000000_0000_000000, required:1, > allocated:0, queryMasterRequest=true, liveWorkers=0 > 2013-08-28 > 11:44:43,460 INFO rm.TajoWorkerResourceManager > (TajoWorkerResourceManager.java:run(196)) - ====> > allocateWorkerResources:eb_0000000000000_0000_000000, required:1, > allocated:0, queryMasterRequest=true, liveWorkers=0 > 2013-08-28 > 11:44:46,460 INFO rm.TajoWorkerResourceManager > (TajoWorkerResourceManager.java:run(196)) - ====> > allocateWorkerResources:eb_0000000000000_0000_000000, required:1, > allocated:0, queryMasterRequest=true, liveWorkers=0 > etc...(it goes on like this forever, until forced exit) > > > Meanwhile, in the terminal I can't see any progress , as I could in the old > version of Tajo. > It doesn't even say > Progress 0, nothing. So far, Tajo only uses Yarn resource manager. In TAJO-127, an additional cluster mode, called standby mode, is added to Tajo system. The standby mode becomes Tajo very fast. So, the standby mode was set as a default cluster mode. But, it makes many people very confuse. So, today, I changed the default mode to on-demand mode using Yarn resource manager. If you merge the recent source to your ingoing work, this problem will be solved immediately > > > 2) > > The > $TAJO_HOME/bin/tajo-daemons.sh doesn't have execution rights and that is > why, when I do $TAJO_HOME/bin/$TAJO_HOME/bin/start-tajo.sh I receive > the following error: > > /home/camelia/tajo_git/incubator-tajo/tajo-dist/target/tajo-0.2.0-SNAPSHOT/bin/start-tajo.sh: > line 35: exec: > /home/camelia/tajo_git/incubator-tajo/tajo-dist/target/tajo-0.2.0-SNAPSHOT/bin/tajo-daemons.sh > > Also, when I do $TAJO_HOME/bin/$TAJO_HOME/bin/stop-tajo.sh I receive the > following error: > > /home/camelia/tajo_git/incubator-tajo/tajo-dist/target/tajo-0.2.0-SNAPSHOT/bin/stop-tajo.sh: > line 34: > /home/camelia/tajo_git/incubator-tajo/tajo-dist/target/tajo-0.2.0-SNAPSHOT/bin/tajo-daemons.sh: > Permission denied > > What is the purpose of this file tajo-daemons.sh ? > I > tried to chmod it for execution but then, when starting or stopping > Tajo it asked for root password each time, which doesn't seem right ... This problem was solved in the master branch. If you synchronise your source tree against the recent source, this problem will be solved. > > > > 3) > An e-mail > received yesterday raised some questions in my mind about the new contents > that tajo-site.xml should have. > Until now I used : > > <configuration> > <property> > <name>tajo.rootdir</name> > <value>file:/home/camelia/tajo_git</value> > </property> > > <property> > <name>tajo.cluster.distributed</name> > <value>true</value> > </property> > </configuration> > > And it worked fine like this. > Is it still all right or do I need to use a different one in the new version > of Tajo? > > Mr Jihoon Son's e-mail suggested : > > <property> > <name>tajo.master.manager.addr</name> > <value>*addr:port*</value> > </property> > This problem was solved in the master branch too. Please sync your source. > > > 4) > > About > the message I received a couple of days ago: "[jira] > [Work started] (TAJO-135) Bump up hadoop to 2.1.0-beta. Work on > TAJO-135 started by Hyunsik Choi". I would like to inform You that my > Hadoop version is Hadoop 0.20.2-cdh3u3 and I will not change it untill > the end of the GSoC project on September 27. I would like to continue > the project with the Hadoop version I have. Tajo should have a release. But, hadoop-2.0.3 is alpha release. Tajo cannot be released with Hadoop alpha version. In addition, the alpha release will be removed from the apache download site soon. So, we have to upgrade Hadoop version inevitably > > > PART B. QUESTIONS ABOUT CODE > > 5) > > The BasicLogicalPlanVisitor can be restricted to a specific query block by > blockName? How? > I need to be able to do this restriction for the outer join rewrite rule I'm > working at. Now, BasicLogicalPlanVisitor only visits the root block. You are right. That feature is very required for queries including multiple query block. I'll expand BasicLogicalPlanVisitor according to this necessary. If you can do this, feel free to create a Jira issue. > > > 6) > > Does the new Tajo version preserve the left deep tree shape of the query plan? > What happens to subqueries: are they unnested and merged in the main query? > > What interests me at this moment regards the types of nodes to be found as > left child and right child of a join node. > > I > worked with the assumption that in the initial un-optimized logical > plan, for a join node: the > right child is always a ScanNode, whereas the left child may be a > ScanNode or a JoinNode. Please contradict me if You consider otherwise. > Yes, you are right. Tajo builds a left deep tree when a logical plan is unoptimised. See visitJoined_table and visitJoined_table_primary methods in SQLAnalyzer.java. The first join tree is created in the parser. If the plan is not optimised, this join tree is preserved in the next phase. > > > 7) > > I understand that FieldEval is for column of a data source, but what is > BinaryEval for? EvalNode is an expression which is used in expression in select list or boolean conditions in where clause. BinaryEval is an binary expression. > > > 8) > > In class FilterPushDownRule , at line 87, the lines are inversed between them: > > LogicalNode left = joinNode.getRightChild(); > LogicalNode right = joinNode.getLeftChild(); > > Shouldn't it be > > LogicalNode left = joinNode.getLeftChild(); > LogicalNode right = joinNode.getRightChild(); > > ? This is a bug. This bug may not cause any problem because FilterPushDown only modifies selection conditions. Nevertheless, it should be fixed. If you find such bugs, feel free to create a Jira issue. > > > Thank You very much in advance and after I finish this part, I'll write to > You again for the physical operators part. > > Yours sincerely, > > Camelia > > > > ________________________________
