Hello Hyunsik, Sorry. I was probably too quick to respond. I have not seen that you have already answered. It is probably still too early. I just got up. This documentation I already know. In this documentation is described the fact that Tajo does not start a new thread when it knows about too small resources, correct? Then I think it's misterious that it still worked a few hours before. I'll try out again in more detail. Many thanks.
Best regards, Chris On 18. September 2014 07:00:39 MESZ, Christian Schwabe <[email protected]> wrote: >Hello Hyunsik, > >thanks anyway. One question is still unanswered. How To configure Tajo >for parallel processing? I already describe in the post before this >one. > >Best regards, >Chris > >On 18. September 2014 06:42:51 MESZ, Hyunsik Choi <[email protected]> >wrote: >>iimissed one thing. I'd like to reserve your dinner when I visit >German >>someday :p >> >>Anyway, thank you for your interest in Tajo. Please keep going your >>trials >>with Tajo. >> >>Thanks! >> >>2014년 9월 18일 목요일, Christian Schwabe<[email protected]>님이 작성한 >>메시지: >> >>> >>> Hello together, >>> >>> I am speechless. The query has been successfully completed. Running >>time: 30 >>> minutes. >>> >>> I've done the following: >>> git pull >>> git clean -d -f >>> git reset --hard >>> patch -p0 < TAJO-1049_3.patch >>> >>> You are great. I'm very thankful. >>> If you would live in Germany, I would invite you to dinner, all of >>your >>> team :-D >>> >>> It is now important for me to understand yet what is there now >>happens in >>> the background. In https://issues.apache.org/jira/browse/TAJO-1048 >>Fixed >>> the bug that the session variables have some parameters are not >>> overwritten at run time, right? >>> Difficult with the understanding it is at >>> https://issues.apache.org/jira/browse/TAJO-1049th What is the >>limiting >>> factor you have fixed it, which now apparently had this effect? >>> >>> In the just traversed test I noticed that only one task is >processed. >>I >>> started just another run with the following configuration in >>tajo-site.xml >>> to use multiple cores: >>> >>> <configuration> >>> <property> >>> <name>tajo.resource-tracker.heartbeat.timeout-secs</name> >>> <value>240000</value> >>> </property> >>> <property> >>> >><name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name> >>> <value>4</value> >>> </property> >>> </configuration> >>> >>> But the tasks are apparently still only processed serially by one >>thread. >>> Is this still an incorrect configuration due? Before the error was >>not >>> fixed, I had made the same settings and determines the tasks were >>> distributed on three cores with four selected cores. >>> >>> I would like to highlight particularly once: I am very grateful to >>you. I >>> very much hope that other interested persons in Tajo from our >>discussions >>> and the experience gained in this and also can learn from it. >>> >>> Warm regards and a thankful, >>> Chris >>> >>> Am 17.09.2014 20:04:21, schrieb Hyunsik Choi: >>> >>> You don't need to apply TAJO-983.140916.patch. As I mentioned, this >>> patch is still ongoing. You only need TAJO-1049_3.patch. >>> >>> Thanks, >>> Hyunsik >>> >>> >>> On Thu, Sep 18, 2014 at 2:45 AM, Christian Schwabe >>> <[email protected] <#14885373167cb354_>> wrote: >>> >>> Hello Hyunsik, >>> >>> Done. Same errors. >>> No compilation possible. >>> >>> Warm regards, >>> Chris >>> >>> >>> Am 17.09.2014 um 19:05 schrieb Hyunsik Choi <[email protected] >>> <#14885373167cb354_>>: >>> >>> Did you update and clean your git repository? In order to apply the >>> submitted patches without conflict, you should use the latest and >>> clean revision. >>> >>> In order to clean and update your revision, please type the >following >>> commands: >>> >>> git clean -d -f >>> git reset --hard >>> patch -p0 < filename.patch >>> >>> >>> Best regards, >>> Hyunsik >>> >>> On Thu, Sep 18, 2014 at 1:57 AM, Christian Schwabe >>> <[email protected] <#14885373167cb354_>> wrote: >>> >>> Hello Hyunsik, >>> >>> i read the bugs to this patch under the review. >>> What i’ve done: >>> -pull to the latest review >>> -I applied the following patches: >>> >>> patch -p0 <TAJO-1049_3.patch >>> patch -p0 <TAJO-983.140916.patch >>> >>> get the following output: >>> >>> >>> christians-mbp:tajo chris$ patch -p0 <TAJO-983.140916.patch >>> patching file >>tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java >>> Hunk #2 succeeded at 52 (offset -1 lines). >>> Hunk #3 FAILED at 68. >>> 1 out of 6 hunks FAILED -- saving rejects to file >>> tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java.rej >>> patching file >>tajo-core/src/main/java/org/apache/tajo/worker/Task.java >>> Hunk #1 FAILED at 48. >>> Hunk #2 succeeded at 97 (offset 1 line). >>> Hunk #3 succeeded at 197 (offset 1 line). >>> Hunk #4 succeeded at 594 (offset 1 line). >>> Hunk #5 succeeded at 639 (offset 1 line). >>> Hunk #6 FAILED at 699. >>> Hunk #7 succeeded at 754 (offset 2 lines). >>> 2 out of 7 hunks FAILED -- saving rejects to file >>> tajo-core/src/main/java/org/apache/tajo/worker/Task.java.rej >>> patching file >>> >>> >>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java >>> patching file >>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java >>> Hunk #2 FAILED at 94. >>> Hunk #3 FAILED at 140. >>> Hunk #4 FAILED at 170. >>> Hunk #5 FAILED at 204. >>> Hunk #6 FAILED at 220. >>> 5 out of 6 hunks FAILED -- saving rejects to file >>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java.rej >>> patching file >>> >>> >>tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java >>> patching file >>> >>> >>tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java >>> christians-mbp:tajo chris$ patch -p0 <TAJO-1049_3.patch >>> patching file >>> >>tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java >>> >>> Patchs seems to be applied correctly, despite the error. >>> >>> recompiled code with mvn clean package -DskipTests -Pdist -Dtar >>> -Dhadoop.version=2.4.0 and get the following output — see attached >>log from >>> maven. Build process crashed. >>> >>> There seems to be something wrong. I hope i’ve done all right. >>> >>> >>> >>> >>> >>> Warm regards, >>> Chris >>> >>> >>> >>> >>> Am 17.09.2014 um 16:39 schrieb Hyunsik Choi <[email protected] >>> <#14885373167cb354_>>: >>> >>> Hi Chris, >>> >>> Do you mean that your query is finished well? Good to hear. But, you >>> need to know that the patch still has some bugs. Please see the >>review >>> https://reviews.apache.org/r/25242/. >>> >>> If your problem is solved by this patch, your problem was related to >>> Fetcher. I'm also expecting your problem is related to this issue >>> (https://github.com/apache/tajo/pull/141/files), which already >>> committed to master branch. You can use this fix from master branch >>> too. >>> >>> Please refer to this document in order to learn how to apply patch. >>We >>> mostly use '--no-prefix' option in order to generate patch files. >So, >>> you should use the patch command with -p0 instead of -p1 as follows: >>> >>> patch -p0 < ${PATCH_FILE}.patch >>> >>> By default, patch command uses -p1. >>> >>> Thanks, >>> Hyunsik >>> >>> On Wed, Sep 17, 2014 at 8:07 AM, Christian Schwabe >>> <[email protected] <#14885373167cb354_>> wrote: >>> >>> Hello guys, >>> >>> after days of efforts to apply the patch from May Hai, I finally be >>able to >>> announce a success today. I want to a brief description where the >>problem >>> was. The few guides out there to apply a patch, using the Git >clients >>often >>> use git apply. However, I had no success. See discussion below. >>> >>> christians-mbp:tajo chris$ git apply --check >>> /Users/chris/Desktop/TAJO-983.140916.patch >>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such >>file or >>> directory >>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file >>or >>> directory >>> error: >>> >>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No >such >>file >>> or directory >>> error: >>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: >>> No such file or directory >>> error: >>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> >>> ########################################### >>> >>> christians-mbp:tajo chris$ git apply --check >>> /Users/chris/Desktop/TAJO-983.140916.patch >>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such >>file or >>> directory >>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file >>or >>> directory >>> error: >>> >>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No >such >>file >>> or directory >>> error: >>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: >>> No such file or directory >>> error: >>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> >>> ########################################### >>> >>> christians-mbp:tajo chris$ git apply -v --check >>> /Users/chris/Desktop/TAJO-983.140916.patch >>> Checking patch src/main/java/org/apache/tajo/worker/Fetcher.java... >>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such >>file or >>> directory >>> Checking patch src/main/java/org/apache/tajo/worker/Task.java... >>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file >>or >>> directory >>> Checking patch >>> >>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java... >>> error: >>> >>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> Checking patch >>src/test/java/org/apache/tajo/worker/TestFetcher.java... >>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No >such >>file >>> or directory >>> Checking patch >>> >>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java... >>> error: >>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java: >>> No such file or directory >>> Checking patch >>> src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java... >>> error: >>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java: >>> No >>> such file or directory >>> >>> ########################################### >>> >>> christians-mbp:tajo-core chris$ git apply >>> /Users/chris/Desktop/TAJO-983.140916.patch >>> /Users/chris/Desktop/TAJO-983.140916.patch:58: trailing whitespace. >>> >>> >>> >>> /Users/chris/Desktop/TAJO-983.140916.patch:80: trailing whitespace. >>> >>> >>> >>> /Users/chris/Desktop/TAJO-983.140916.patch:487: trailing whitespace. >>> return " (start=" + startOffset() + ", length=" + length + ", >>> fromRemote=" + fromRemote >>> warning: 3 lines add whitespace errors. >>> >>> ########################################### >>> >>> Removal of whitespaces also brought no success. >>> My hope depended least on this patch. As far Tajo not run in my >>environment >>> and my standard configuration, I will unfortunately have to complete >>my >>> thesis with the result that it is currently not suitable for >>practical use. >>> This really is not a reproach to formulate that clear again. I >really >>do >>> appreciate your hard work very much. However, I also think it's >>slightly >>> doubtful that the advice you have given can't read anywhere and only >>> inquired. I think that's still a pity. Surely you can not really >>expect, or >>> rather the can but not really your goal to be always be the first >>point of >>> contact. I think at this point there is still a lot of catching up >to >>do. >>> This is still a large deficit. Surely you can not really expect, or >>rather >>> it is hopefully not really your goal to be always be the first point >>of >>> contact. I think at this point there is still a lot of catching up >to >>do. >>> >>> I am not come within the scope of my thesis within the time to the >>desired >>> goal, but of course I'm still interested in a close contact with you >>and >>> and >>> a further effort to solve the outstanding problem. >>> >>> Until I have encountered the following with the patch was >>successfully >>> carried out. Unfortunately, this patch did not bring the hoped >>solution. No >>> change in the behavior described herein. Are there any other ideas >>that I >>> could try to fix the problem? >>> >>> I really thank you very much for your support thus far. I really >>thank you >>> very much for your support thus far. I really thank you very much >for >>your >>> support thus far. I really thank you very much for your support thus >>far. I >>> really thank you very much for your support thus far. I really thank >>you >>> very much for your support thus far. I really thank you very much >for >>your >>> support thus far. I really thank you very much for your support thus >>far. >>> ;-) >>> >>> >>> christians-mbp:tajo chris$ cd tajo-yarn-pullserver/ >>> christians-mbp:tajo-yarn-pullserver chris$ patch -p1 < >>> ../TAJO-983.140916.patch >>> can't find file to patch at input line 5 >>> Perhaps you used the wrong -p or --strip option? >>> The text leading up to this was: >>> -------------------------- >>> |diff --git >>tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java >>> tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java >>> |index 64475fe..7310736 100644 >>> |--- tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java >>> |+++ tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java >>> -------------------------- >>> File to patch: >>> Skip this patch? [y] y >>> Skipping patch. >>> 6 out of 6 hunks ignored >>> can't find file to patch at input line 98 >>> Perhaps you used the wrong -p or --strip option? >>> The text leading up to this was: >>> -------------------------- >>> |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Task.java >>> tajo-core/src/main/java/org/apache/tajo/worker/Task.java >>> |index 7b4cbe1..ecc4078 100644 >>> |--- tajo-core/src/main/java/org/apache/tajo/worker/Task.java >>> |+++ tajo-core/src/main/java/org/apache/tajo/worker/Task.java >>> -------------------------- >>> File to patch: y >>> y: No such file or directory >>> Skip this patch? [y] y >>> Skipping patch. >>> 7 out of 7 hunks ignored >>> can't find file to patch at input line 339 >>> Perhaps you used the wrong -p or --strip option? >>> The text leading up to this was: >>> -------------------------- >>> |diff --git >>> >>> >>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java >>> >>> >>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java >>> |index 4f11168..911c0c2 100644 >>> |--- >>> >>> >>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java >>> |+++ >>> >>> >>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java >>> -------------------------- >>> File to patch: y >>> y: No such file or directory >>> Skip this patch? [y] y >>> Skipping patch. >>> 1 out of 1 hunk ignored >>> can't find file to patch at input line 362 >>> Perhaps you used the wrong -p or --strip option? >>> The text leading up to this was: >>> -------------------------- >>> |diff --git >>tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java >>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java >>> |index b755e02..cb19d54 100644 >>> |--- tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java >>> |+++ tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java >>> -------------------------- >>> File to patch: y >>> y: No such file or directory >>> Skip this patch? [y] y >>> Skipping patch. >>> 6 out of 6 hunks ignored >>> patching file >>> src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java >>> patching file >>> src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java >>> christians-mbp:tajo-yarn-pullserver chris$ pwd >>> /Users/chris/6_Semester/Thesis/07_git/tajo/tajo-yarn-pullserver >>> >>> >>> Warm regards, >>> Chris >>> >>> >>> >>> >>> >>>
