I'm very happy to hear your success. Your problem was caused by one trivial bug and one old limitation. your understanding on TAJO-1048 is right. TAJO-1049 is a limit of parallel degree. it was limited by the maximum cluster capacity because of followings: - the number of parallel degree affects the number of shuffle files. - As the number of parallel degree, the num of files increases too. - Too many files can cause various problems and performance degradation.
If you want to increase the cluster capacity, please see the manual. it contains some example :) http://tajo.apache.org/docs/current/configuration/worker_configuration.html#maximum-number-of-parallel-running-tasks-for-each-worker best regards, hyunsik 2014년 9월 18일 목요일, Christian Schwabe<[email protected]>님이 작성한 메시지: > > Hello together, > > I am speechless. The query has been successfully completed. Running time: 30 > minutes. > > I've done the following: > git pull > git clean -d -f > git reset --hard > patch -p0 < TAJO-1049_3.patch > > You are great. I'm very thankful. > If you would live in Germany, I would invite you to dinner, all of your > team :-D > > It is now important for me to understand yet what is there now happens in > the background. In https://issues.apache.org/jira/browse/TAJO-1048 Fixed > the bug that the session variables have some parameters are not > overwritten at run time, right? > Difficult with the understanding it is at > https://issues.apache.org/jira/browse/TAJO-1049th What is the limiting > factor you have fixed it, which now apparently had this effect? > > In the just traversed test I noticed that only one task is processed. I > started just another run with the following configuration in tajo-site.xml > to use multiple cores: > > <configuration> > <property> > <name>tajo.resource-tracker.heartbeat.timeout-secs</name> > <value>240000</value> > </property> > <property> > <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name> > <value>4</value> > </property> > </configuration> > > But the tasks are apparently still only processed serially by one thread. > Is this still an incorrect configuration due? Before the error was not > fixed, I had made the same settings and determines the tasks were > distributed on three cores with four selected cores. > > I would like to highlight particularly once: I am very grateful to you. I > very much hope that other interested persons in Tajo from our discussions > and the experience gained in this and also can learn from it. > > Warm regards and a thankful, > Chris > > Am 17.09.2014 20:04:21, schrieb Hyunsik Choi: > > You don't need to apply TAJO-983.140916.patch. As I mentioned, this > patch is still ongoing. You only need TAJO-1049_3.patch. > > Thanks, > Hyunsik > > > On Thu, Sep 18, 2014 at 2:45 AM, Christian Schwabe > <[email protected] <#14885373167cb354_>> wrote: > > Hello Hyunsik, > > Done. Same errors. > No compilation possible. > > Warm regards, > Chris > > > Am 17.09.2014 um 19:05 schrieb Hyunsik Choi <[email protected] > <#14885373167cb354_>>: > > Did you update and clean your git repository? In order to apply the > submitted patches without conflict, you should use the latest and > clean revision. > > In order to clean and update your revision, please type the following > commands: > > git clean -d -f > git reset --hard > patch -p0 < filename.patch > > > Best regards, > Hyunsik > > On Thu, Sep 18, 2014 at 1:57 AM, Christian Schwabe > <[email protected] <#14885373167cb354_>> wrote: > > Hello Hyunsik, > > i read the bugs to this patch under the review. > What i’ve done: > -pull to the latest review > -I applied the following patches: > > patch -p0 <TAJO-1049_3.patch > patch -p0 <TAJO-983.140916.patch > > get the following output: > > > christians-mbp:tajo chris$ patch -p0 <TAJO-983.140916.patch > patching file tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java > Hunk #2 succeeded at 52 (offset -1 lines). > Hunk #3 FAILED at 68. > 1 out of 6 hunks FAILED -- saving rejects to file > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java.rej > patching file tajo-core/src/main/java/org/apache/tajo/worker/Task.java > Hunk #1 FAILED at 48. > Hunk #2 succeeded at 97 (offset 1 line). > Hunk #3 succeeded at 197 (offset 1 line). > Hunk #4 succeeded at 594 (offset 1 line). > Hunk #5 succeeded at 639 (offset 1 line). > Hunk #6 FAILED at 699. > Hunk #7 succeeded at 754 (offset 2 lines). > 2 out of 7 hunks FAILED -- saving rejects to file > tajo-core/src/main/java/org/apache/tajo/worker/Task.java.rej > patching file > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java > patching file > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java > Hunk #2 FAILED at 94. > Hunk #3 FAILED at 140. > Hunk #4 FAILED at 170. > Hunk #5 FAILED at 204. > Hunk #6 FAILED at 220. > 5 out of 6 hunks FAILED -- saving rejects to file > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java.rej > patching file > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > patching file > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > christians-mbp:tajo chris$ patch -p0 <TAJO-1049_3.patch > patching file > tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java > > Patchs seems to be applied correctly, despite the error. > > recompiled code with mvn clean package -DskipTests -Pdist -Dtar > -Dhadoop.version=2.4.0 and get the following output — see attached log from > maven. Build process crashed. > > There seems to be something wrong. I hope i’ve done all right. > > > > > > Warm regards, > Chris > > > > > Am 17.09.2014 um 16:39 schrieb Hyunsik Choi <[email protected] > <#14885373167cb354_>>: > > Hi Chris, > > Do you mean that your query is finished well? Good to hear. But, you > need to know that the patch still has some bugs. Please see the review > https://reviews.apache.org/r/25242/. > > If your problem is solved by this patch, your problem was related to > Fetcher. I'm also expecting your problem is related to this issue > (https://github.com/apache/tajo/pull/141/files), which already > committed to master branch. You can use this fix from master branch > too. > > Please refer to this document in order to learn how to apply patch. We > mostly use '--no-prefix' option in order to generate patch files. So, > you should use the patch command with -p0 instead of -p1 as follows: > > patch -p0 < ${PATCH_FILE}.patch > > By default, patch command uses -p1. > > Thanks, > Hyunsik > > On Wed, Sep 17, 2014 at 8:07 AM, Christian Schwabe > <[email protected] <#14885373167cb354_>> wrote: > > Hello guys, > > after days of efforts to apply the patch from May Hai, I finally be able to > announce a success today. I want to a brief description where the problem > was. The few guides out there to apply a patch, using the Git clients often > use git apply. However, I had no success. See discussion below. > > christians-mbp:tajo chris$ git apply --check > /Users/chris/Desktop/TAJO-983.140916.patch > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file or > directory > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or > directory > error: > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: > No > such file or directory > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such file > or directory > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: > No such file or directory > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: > No > such file or directory > > ########################################### > > christians-mbp:tajo chris$ git apply --check > /Users/chris/Desktop/TAJO-983.140916.patch > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file or > directory > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or > directory > error: > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: > No > such file or directory > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such file > or directory > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: > No such file or directory > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: > No > such file or directory > > ########################################### > > christians-mbp:tajo chris$ git apply -v --check > /Users/chris/Desktop/TAJO-983.140916.patch > Checking patch src/main/java/org/apache/tajo/worker/Fetcher.java... > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file or > directory > Checking patch src/main/java/org/apache/tajo/worker/Task.java... > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or > directory > Checking patch > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java... > error: > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: > No > such file or directory > Checking patch src/test/java/org/apache/tajo/worker/TestFetcher.java... > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such file > or directory > Checking patch > src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java... > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: > No such file or directory > Checking patch > src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java... > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: > No > such file or directory > > ########################################### > > christians-mbp:tajo-core chris$ git apply > /Users/chris/Desktop/TAJO-983.140916.patch > /Users/chris/Desktop/TAJO-983.140916.patch:58: trailing whitespace. > > > > /Users/chris/Desktop/TAJO-983.140916.patch:80: trailing whitespace. > > > > /Users/chris/Desktop/TAJO-983.140916.patch:487: trailing whitespace. > return " (start=" + startOffset() + ", length=" + length + ", > fromRemote=" + fromRemote > warning: 3 lines add whitespace errors. > > ########################################### > > Removal of whitespaces also brought no success. > My hope depended least on this patch. As far Tajo not run in my environment > and my standard configuration, I will unfortunately have to complete my > thesis with the result that it is currently not suitable for practical use. > This really is not a reproach to formulate that clear again. I really do > appreciate your hard work very much. However, I also think it's slightly > doubtful that the advice you have given can't read anywhere and only > inquired. I think that's still a pity. Surely you can not really expect, or > rather the can but not really your goal to be always be the first point of > contact. I think at this point there is still a lot of catching up to do. > This is still a large deficit. Surely you can not really expect, or rather > it is hopefully not really your goal to be always be the first point of > contact. I think at this point there is still a lot of catching up to do. > > I am not come within the scope of my thesis within the time to the desired > goal, but of course I'm still interested in a close contact with you and > and > a further effort to solve the outstanding problem. > > Until I have encountered the following with the patch was successfully > carried out. Unfortunately, this patch did not bring the hoped solution. No > change in the behavior described herein. Are there any other ideas that I > could try to fix the problem? > > I really thank you very much for your support thus far. I really thank you > very much for your support thus far. I really thank you very much for your > support thus far. I really thank you very much for your support thus far. I > really thank you very much for your support thus far. I really thank you > very much for your support thus far. I really thank you very much for your > support thus far. I really thank you very much for your support thus far. > ;-) > > > christians-mbp:tajo chris$ cd tajo-yarn-pullserver/ > christians-mbp:tajo-yarn-pullserver chris$ patch -p1 < > ../TAJO-983.140916.patch > can't find file to patch at input line 5 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java > |index 64475fe..7310736 100644 > |--- tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java > |+++ tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java > -------------------------- > File to patch: > Skip this patch? [y] y > Skipping patch. > 6 out of 6 hunks ignored > can't find file to patch at input line 98 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Task.java > tajo-core/src/main/java/org/apache/tajo/worker/Task.java > |index 7b4cbe1..ecc4078 100644 > |--- tajo-core/src/main/java/org/apache/tajo/worker/Task.java > |+++ tajo-core/src/main/java/org/apache/tajo/worker/Task.java > -------------------------- > File to patch: y > y: No such file or directory > Skip this patch? [y] y > Skipping patch. > 7 out of 7 hunks ignored > can't find file to patch at input line 339 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |diff --git > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java > |index 4f11168..911c0c2 100644 > |--- > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java > |+++ > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java > -------------------------- > File to patch: y > y: No such file or directory > Skip this patch? [y] y > Skipping patch. > 1 out of 1 hunk ignored > can't find file to patch at input line 362 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |diff --git tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java > |index b755e02..cb19d54 100644 > |--- tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java > |+++ tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java > -------------------------- > File to patch: y > y: No such file or directory > Skip this patch? [y] y > Skipping patch. > 6 out of 6 hunks ignored > patching file > src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > patching file > src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > christians-mbp:tajo-yarn-pullserver chris$ pwd > /Users/chris/6_Semester/Thesis/07_git/tajo/tajo-yarn-pullserver > > > Warm regards, > Chris > > > > > >
