Thank you Mai for your comment.
On Fri, Sep 12, 2014 at 9:26 AM, Thanh Mai <[email protected]> wrote: > Hi Christian, > > I attached in this email a patch file for a JIRA issue > (https://issues.apache.org/jira/browse/TAJO-983). I think that you can try > to apply this patch to your TAJO source base. Although I made this patch for > a different problem (and the patch is still under review), it might help a > little bit because this patch reduces the data transmission. Since you are > joining relatively big tables, there will be many data chunks transferred > around. As a result, this patch can improve the processing speed. Anyway, > you can revert to the stable code base anytime if you do not like the patch. > > Beside, when you compile and install TAJO, be sure include the flag > "-Dhadoop.version=2.4.0". About 2 weeks ago, I also got a never-ending-query > problem due to a bug in Pull Server, which seems to be caused by Hadoop > native library. With the "-Dhadoop.version=2.4.0" flag, the problem > disappears. > > Sincerely, > > Mai Hai Thanh > > > On Fri, Sep 12, 2014 at 6:52 AM, Christian Schwabe > <[email protected]> wrote: >> >> Hi Hyunsik, >> >> thank you for your assistance. >> Sorry, your suggestions have been unsuccessful. And I have several sizes >> of parameters used and started some test runs. Every time it ran out and the >> nested loop join is selected. >> Here are the following used set statements: >> >> 1) \set OUTER_HASH_JOIN_SIZE_LIMIT 1024 >> 2) \set OUTER_HASH_JOIN_SIZE_LIMIT 4096 >> >> 3) \set JOIN_TASK_INPUT_SIZE 64 >> 3.1) \set JOIN_PER_SHUFFLE_SIZE 64 >> >> 4) \set JOIN_TASK_INPUT_SIZE 32 >> 4.1) \set JOIN_PER_SHUFFLE_SIZE 32 >> >> The biggest table in parquet format is 1GB. >> If I understand correctly, it should over-large, but sufficient to put >> OUTER_HASH_JOIN_SIZE_LIMIT to the value 4096, right? >> >> So many problems. I can honestly not understand and comprehend. What I'm >> doing here is nothing extraordinary?! >> >> Best regards, >> a sad Chris >> >> >> Am 11.09.2014 um 17:14 schrieb Hyunsik Choi <[email protected]>: >> >> > Thank you Chris for sharing detailed information. I can reproduce the >> > problem. >> > >> > The plan choice algorithm is working as following procedure: >> > 1. If the smallest table of both join tables is less than some >> > threshold (OUTER_HASH_JOIN_SIZE_LIMIT), it chooses hash left outer >> > join algorithm. >> > 2. Otherwise, it chooses nested loop left outer join algorithm. >> > >> > Your problem was that your smallest table size exceeds >> > OUTER_HASH_JOIN_SIZE_LIMIT. There are various solutions you can try. >> > >> > First of all, you can try to directly increase >> > OUTER_HASH_JOIN_SIZE_LIMIT. Please try set command to increase the >> > threshold (megabytes unit) in tsql as follows: >> > >> > tajo> \set OUTER_HASH_JOIN_SIZE_LIMIT 256 >> > >> > Instead of 256MB, you can use proper number according to your machine. >> > But, this solution can be failed if available memory is smaller than >> > the smallest table. >> > >> > The better solution is that you decrease JOIN_TASK_INPUT_SIZE and >> > JOIN_PER_SHUFFLE_SIZE. Their default sizes are 128 MB. If you decrease >> > them, each join task will be smaller. As a result, smallest table will >> > be fit to current hash join threshold. For it, please type the set >> > commands as follows: >> > >> > tajo> \set JOIN_TASK_INPUT_SIZE 64 >> > tajo> \set JOIN_PER_SHUFFLE_SIZE 64 >> > >> > >> > Thanks, >> > Hyunsik >> > >> > >> > On Thu, Sep 11, 2014 at 7:20 PM, Christian Schwabe >> > <[email protected]> wrote: >> >> Hi Hyunsik, >> >> >> >> >> >> thanks for your response and to take this problem immediately. Many >> >> thanks. >> >> Attached the QueryPlan for my problematic Query. >> >> If you need any further information feel free to ask. >> >> Step ‚eb_1410429199927_0001_000005‘ is the problematic step. >> >> >> >> >> >> Warm regards, >> >> Chris >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Am 11.09.2014 um 04:21 schrieb Hyunsik Choi <[email protected]>: >> >> >> >> Hi Chris, >> >> >> >> Could you share the plan of your problematic query? I'm trying to >> >> reproduce >> >> your problem, but the current planner of my system does not choose >> >> nested >> >> loop join. So, I'd like to see your query plan. The query plan will be >> >> shown >> >> in the query detail page in web UI. >> >> >> >> Usually, this kind of bugs can be solved for few hours if I reproduce. >> >> >> >> Best regards, >> >> Hyunsik >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Sep 11, 2014 at 1:54 AM, Hyunsik Choi <[email protected]> >> >> wrote: >> >>> >> >>> Hi Chris, >> >>> >> >>> I was on vacation due to holiday. Tomorrow, I'll response how long >> >>> will it >> >>> take to solve the problem after I dig it more. >> >>> >> >>> Best regards, >> >>> Hyunsik >> >>> >> >>> On Wed, Sep 10, 2014 at 8:07 PM, Christian Schwabe >> >>> <[email protected]> wrote: >> >>>> >> >>>> Hello Hyunsik, >> >>>> >> >>>> are there any already saved new knowledge? Approaching the time where >> >>>> I >> >>>> need to finish my thesis is namely to end. And so far it was not >> >>>> possible to >> >>>> use Tajo within the scope for which it was originally intended. When >> >>>> changes >> >>>> to it in the near future not run out my work on it I will make a >> >>>> recommendation currently not taking this software in production use. >> >>>> That >> >>>> would be bad, because I did not expect with this end. >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> >> >>>> >> >>>> Am 06.09.2014 14:01:41, schrieb Hyunsik Choi: >> >>>> >> >>>> Thank you for sharing your query. I'll dig into this problem. I'll >> >>>> response soon. >> >>>> >> >>>> Best regards, >> >>>> Hyunsik >> >>>> >> >>>> >> >>>> On Fri, Sep 5, 2014 at 11:33 PM, Christian Schwabe >> >>>> <[email protected]> wrote: >> >>>> >> >>>> >> >>>> Hello Hyunsik, >> >>>> >> >>>> thanks for your response. >> >>>> Here is the anonymized query. >> >>>> If you need any further information feel free to ask ;) >> >>>> >> >>>>>> QUERY: >> >>>> drop table if exists anonym1; >> >>>> create table anonym1 using parquet >> >>>> as >> >>>> select >> >>>> table1.a, >> >>>> table1.b, >> >>>> table1.c, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.ai >> >>>> else >> >>>> table1.ai end as ai, >> >>>> table1.d, >> >>>> table1.e, >> >>>> table1.f, >> >>>> table1.g, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.aj >> >>>> else table1.aj end as aj, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.ak >> >>>> else table1.ak end as ak, >> >>>> table1.h, >> >>>> table1.i, >> >>>> table1.j, >> >>>> table1.k, >> >>>> table1.l, >> >>>> table1.m, >> >>>> table1.n, >> >>>> table1.o, >> >>>> table1.p, >> >>>> table1.q, >> >>>> table1.r, >> >>>> table1.s, >> >>>> table1.t, >> >>>> table1.u, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.ag >> >>>> else table1.ag end as ag, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.al >> >>>> else table1.al end as al, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.am >> >>>> else table1.am end as am, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.an >> >>>> else table1.an end as an, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.ao >> >>>> else table1.ao end as ao, >> >>>> case when table1.ag = '1800-01-01' and table3.ah = '' then table3.ap >> >>>> else table1.ap end as ap, >> >>>> table1.v, >> >>>> table2.w, >> >>>> table2.x, >> >>>> table2.y, >> >>>> table2.z, >> >>>> table2.aa, >> >>>> table2.ab, >> >>>> table2.ac, >> >>>> table2.ad, >> >>>> table2.ae, >> >>>> table2.af >> >>>> from anonym2 table1 >> >>>> left join dfkkopw_hist table3 on ( >> >>>> table1.a = table3.a >> >>>> and table1.b = table3.b >> >>>> and table1.c = table3.c >> >>>> and table1.aq = table3.aq >> >>>> ) >> >>>> inner join anonym3 table2 on ( >> >>>> table1.a = table2.a >> >>>> and table1.b = table2.b >> >>>> and table1.c = table2.c >> >>>> ) >> >>>> ; >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> Warm regards, >> >>>> Chris >> >>>> >> >>>> Am 05.09.2014 13:46:44, schrieb Hyunsik Choi: >> >>>> >> >>>> Hi Chris, >> >>>> >> >>>> That's ok. I'd like suggest the following manners. I've been used >> >>>> those >> >>>> manners to fix special user query cases. I'm expecting that you can >> >>>> do at >> >>>> least one of them. >> >>>> >> >>>> - The first manner is to change column names and table names in order >> >>>> to >> >>>> hide your business logic if possible. >> >>>> - The second manner is to share only join conditions with changing >> >>>> field >> >>>> names. >> >>>> >> >>>> Best regards, >> >>>> Hyunsik >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Fri, Sep 5, 2014 at 3:15 AM, Christian Schwabe >> >>>> <[email protected]> wrote: >> >>>> >> >>>> Hello Hyunsik, >> >>>> >> >>>> are the more detailed information been helpful? I could offer you >> >>>> attend >> >>>> a TeamViewer session to examine the problem more closely. Would that >> >>>> be a >> >>>> proposal on which you would let you in? So at least ensured that I >> >>>> must not >> >>>> publish queries. >> >>>> Due to the time difference in Germany would have to find only a >> >>>> suitable >> >>>> date and time. >> >>>> >> >>>> >> >>>> Warm regards, >> >>>> Chris >> >>>> >> >>>> >> >>>> Am 01.09.2014 20:26:35, schrieb Christian Schwabe: >> >>>> >> >>>> Hello Hyunsik, >> >>>> >> >>>> the late reply does not matter. I'm glad you support me so much. >> >>>> That's >> >>>> worth a lot! Unfortunately I can not post the query for privacy >> >>>> reasons. Can >> >>>> I send you other information to identify the problem in more detail? >> >>>> >> >>>> I can try to describe the general query: The projection refers to 43 >> >>>> fields, it is a left join combined with an inner join, the join >> >>>> conditions >> >>>> are made for up to four fields. >> >>>> The tables are saved in parquet-format. >> >>>> First table (12,853,555 rows 544,6MB) >> >>>> Second table (17,896,966 rows 361,1MB) >> >>>> Third table (18,825,153 rows 943,2MB) >> >>>> Is there a way to force Tajo to a specific algorithm? >> >>>> >> >>>> You've talked about to give another advice, in the case of the nested >> >>>> join is the only choice. What would this mean excactly? >> >>>> >> >>>> Warm regards, >> >>>> Chris >> >>>> >> >>>> Am 01.09.2014 19:09:59, schrieb Hyunsik Choi: >> >>>> >> >>>> Hi Chris, >> >>>> >> >>>> Could you share your query? I can see a part of your query from your >> >>>> screenshot, but the query is not entire one. >> >>>> >> >>>> I think that the first problem is that the query planner chooses >> >>>> nested >> >>>> loop join which is very inefficient join algorithm. In some cases, >> >>>> there is >> >>>> no alternative if there is no join condition or join condition is >> >>>> theta >> >>>> join. But, I'm suspecting that it may be a bug because most of joins >> >>>> are >> >>>> equi-joins. >> >>>> >> >>>> I'll give another advice If block nested-loop join is only choice. >> >>>> >> >>>> Best regards, >> >>>> Hyunsik >> >>>> >> >>>> >> >>>> On Mon, Sep 1, 2014 at 8:54 PM, Christian Schwabe >> >>>> <[email protected]> wrote: >> >>>> >> >>>> >> >>>> Hello guys, >> >>>> >> >>>> Does anyone have an idea what I could do against the failure of a >> >>>> "nested >> >>>> loop join"? >> >>>> >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> Am 27.08.2014 10:55:27, schrieb Christian Schwabe: >> >>>> >> >>>> Hello Jinhang, >> >>>> >> >>>> Thank you for your helpful hints. In fact, he makes an impact. You >> >>>> can >> >>>> see it on the attached screenshot. Now he seems to hang when merging >> >>>> the two >> >>>> tables. This helpful hint you should definitely include in your >> >>>> documentation. I have previously read this parameter still nowhere … >> >>>> >> >>>> The log show the following (partial list): >> >>>> >> >>>> >> >>>> 2014-08-27 10:13:09,583 INFO org.apache.tajo.worker.Task: * Local >> >>>> task >> >>>> dir: file:/tmp/tajo-chris/tmpdir/q_1409126149411_0001/output/5/1_0 >> >>>> 2014-08-27 10:13:09,583 INFO org.apache.tajo.worker.Task: >> >>>> ================================== >> >>>> 2014-08-27 10:13:09,584 INFO org.apache.tajo.worker.Task: the >> >>>> directory >> >>>> is created >> >>>> >> >>>> file:/tmp/tajo-chris/tmpdir/q_1409126149411_0001/in/eb_1409126149411_0001_000005/1/0/eb_1409126149411_0001_000003 >> >>>> 2014-08-27 10:13:09,584 INFO org.apache.tajo.worker.Task: the >> >>>> directory >> >>>> is created >> >>>> >> >>>> file:/tmp/tajo-chris/tmpdir/q_1409126149411_0001/in/eb_1409126149411_0001_000005/1/0/eb_1409126149411_0001_000004 >> >>>> 2014-08-27 10:13:09,600 INFO >> >>>> org.apache.tajo.worker.TaskAttemptContext: >> >>>> Query status of ta_1409126149411_0001_000005_000001_00 is changed to >> >>>> TA_RUNNING >> >>>> 2014-08-27 10:13:09,602 INFO org.apache.tajo.worker.Fetcher: Status: >> >>>> FETCH_FETCHING, >> >>>> >> >>>> URI:http://192.168.178.101:52334/?qid=q_1409126149411_0001&sid=4&p=1&type=h&ta=68_0,17_0,50_0,32_0,61_0,25_0,55_0,60_0,47_0,69_0,16_0,38_0,2_0,24_0,33_0,11_0,5_0,41_0,44_0,66_0,52_0,19_0,8_0,30_0,63_0,27_0,21_0,26_0,72_0,67_0,36_0,57_0,18_0,49_0,31_0,13_0,29_0,62_0,10_0,0_0,46_0,65_0,3_0,1_0,54_0,39_0,15_0,37_0,23_0,6_0,64_0,59_0,73_0,51_0,28_0,42_0,34_0,12_0,20_0,9_0,56_0,35_0,45_0,48_0,53_0,4_0,70_0,71_0,58_0,40_0,22_0,7_0,14_0,43_0 >> >>>> 2014-08-27 10:13:09,602 INFO org.apache.tajo.worker.Fetcher: Status: >> >>>> FETCH_FETCHING, >> >>>> >> >>>> URI:http://christians-mbp.fritz.box:52334/?qid=q_1409126149411_0001&sid=3&p=1&type=h&ta=1_0 >> >>>> 2014-08-27 10:13:09,602 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: Current number of >> >>>> shuffle >> >>>> connections (2) >> >>>> 2014-08-27 10:13:09,602 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: Current number of >> >>>> shuffle >> >>>> connections (3) >> >>>> 2014-08-27 10:13:09,603 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: PullServer request >> >>>> param: >> >>>> shuffleType=h, sid=3, partId=1, taskIds=[1_0] >> >>>> 2014-08-27 10:13:09,603 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: PullServer baseDir: >> >>>> /tmp/tajo-chris/tmpdir/q_1409126149411_0001/output >> >>>> 2014-08-27 10:13:09,604 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: PullServer request >> >>>> param: >> >>>> shuffleType=h, sid=4, partId=1, >> >>>> >> >>>> taskIds=[68_0,17_0,50_0,32_0,61_0,25_0,55_0,60_0,47_0,69_0,16_0,38_0,2_0,24_0,33_0,11_0,5_0,41_0,44_0,66_0,52_0,19_0,8_0,30_0,63_0,27_0,21_0,26_0,72_0,67_0,36_0,57_0,18_0,49_0,31_0,13_0,29_0,62_0,10_0,0_0,46_0,65_0,3_0,1_0,54_0,39_0,15_0,37_0,23_0,6_0,64_0,59_0,73_0,51_0,28_0,42_0,34_0,12_0,20_0,9_0,56_0,35_0,45_0,48_0,53_0,4_0,70_0,71_0,58_0,40_0,22_0,7_0,14_0,43_0] >> >>>> 2014-08-27 10:13:09,604 INFO >> >>>> org.apache.tajo.pullserver.TajoPullServerService: PullServer baseDir: >> >>>> /tmp/tajo-chris/tmpdir/q_1409126149411_0001/output >> >>>> 2014-08-27 10:13:15,313 INFO org.apache.tajo.worker.Fetcher: Data >> >>>> fetch >> >>>> is done (total received bytes: 991342257) >> >>>> 2014-08-27 10:13:15,317 INFO org.apache.tajo.worker.Fetcher: Status: >> >>>> FETCH_FINISHED, >> >>>> >> >>>> URI:http://192.168.178.101:52334/?qid=q_1409126149411_0001&sid=4&p=1&type=h&ta=68_0,17_0,50_0,32_0,61_0,25_0,55_0,60_0,47_0,69_0,16_0,38_0,2_0,24_0,33_0,11_0,5_0,41_0,44_0,66_0,52_0,19_0,8_0,30_0,63_0,27_0,21_0,26_0,72_0,67_0,36_0,57_0,18_0,49_0,31_0,13_0,29_0,62_0,10_0,0_0,46_0,65_0,3_0,1_0,54_0,39_0,15_0,37_0,23_0,6_0,64_0,59_0,73_0,51_0,28_0,42_0,34_0,12_0,20_0,9_0,56_0,35_0,45_0,48_0,53_0,4_0,70_0,71_0,58_0,40_0,22_0,7_0,14_0,43_0 >> >>>> 2014-08-27 10:13:18,685 INFO org.apache.tajo.worker.Fetcher: Data >> >>>> fetch >> >>>> is done (total received bytes: 2225872857) >> >>>> 2014-08-27 10:13:18,685 INFO org.apache.tajo.worker.Fetcher: Status: >> >>>> FETCH_FINISHED, >> >>>> >> >>>> URI:http://christians-mbp.fritz.box:52334/?qid=q_1409126149411_0001&sid=3&p=1&type=h&ta=1_0 >> >>>> 2014-08-27 10:13:18,686 INFO org.apache.tajo.worker.Task: >> >>>> ta_1409126149411_0001_000005_000001_00 All fetches are done! >> >>>> 2014-08-27 10:13:18,688 INFO >> >>>> org.apache.tajo.engine.planner.PhysicalPlannerImpl: Left Outer Join >> >>>> (9) >> >>>> chooses [Nested Loop Join]. >> >>>> >> >>>> >> >>>> What means „Nested Loop Join“? The last entry in the above log is >> >>>> already >> >>>> passed since 40 minutes. Nothing happens since that point. >> >>>> >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> >> >>>> Am 26.08.2014 um 08:17 schrieb Jinhang Choi <[email protected]>: >> >>>> >> >>>> At first, you can take a try of changing resource-tracker's heartbeat >> >>>> timeout before investigating the actual problem. worker log shows >> >>>> on-going >> >>>> fetcher operations even though acknowledging heartbeat responses are >> >>>> delayed. >> >>>> >> >>>> besides, master log indicates Worker's deactivation with >> >>>> LivelinessMonitor's timeout as follow: >> >>>> >> >>>> ==== >> >>>> >> >>>> 2014-08-25 21:18:47,968 INFO >> >>>> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: >> >>>> Expired:christians-mbp.fritz.box:28093:28091 Timed out after 120 secs >> >>>> >> >>>> 2014-08-25 21:18:47,978 INFO org.apache.tajo.master.rm.Worker: >> >>>> Deactivating Node christians-mbp.fritz.box:28093:28091 as it is now >> >>>> LOST >> >>>> >> >>>> ==== >> >>>> >> >>>> >> >>>> How about testing tajo.resource-tracker.heartbeat.timeout-secs in >> >>>> tajo-site.xml like this? >> >>>> >> >>>> >> >>>> >> >>>> <configuration> >> >>>> >> >>>> .... >> >>>> >> >>>> <property> >> >>>> >> >>>> <name>tajo.resource-tracker.heartbeat.timeout-secs</name> >> >>>> >> >>>> <value>240000</value> // or, your own longer value. default >> >>>> is >> >>>> 120*1000 (2 minutes) >> >>>> >> >>>> </property> >> >>>> >> >>>> </configuration> >> >>>> >> >>>> >> >>>> >> >>>> Sincerely, >> >>>> >> >>>> ---------------------------------------------- >> >>>> Jinhang Choi, CTO. >> >>>> Linewalks Inc. Seoul 137-860, Korea >> >>>> Office: +82 70 7702 3043 >> >>>> FAX: +82 2 2055 0612 >> >>>> >> >>>> -----Original Message----- >> >>>> From: "Christian Schwabe"<[email protected]> >> >>>> To: <[email protected]>; "Jinhang Choi"<[email protected]>; >> >>>> Cc: >> >>>> Sent: 2014-08-26 (화) 12:09:37 >> >>>> Subject: Re: Big big query >> >>>> >> >>>> Hello everyone, >> >>>> >> >>>> >> >>>> I've tested a lot again today. I want to share my experiences and >> >>>> discuss >> >>>> it here. I have attached again a log of the Worker for the query >> >>>> which runs >> >>>> seemingly endless. >> >>>> As requested by you I have assigned to the worker 4GB. 8GB has my >> >>>> MacBook >> >>>> available. >> >>>> Also assign more memory is no solution. >> >>>> Joins on small tables with windows_functions an little content seem >> >>>> to be >> >>>> no problem. >> >>>> Joins with many columns and large tables, as it is a table in this >> >>>> example seems to be a real problem. My guess at this point is an >> >>>> incorrect >> >>>> memory management of Tajo. >> >>>> I have a video made by you better show the WebUI and to get a better >> >>>> picture of the situation. In combination with the submitted logs I >> >>>> hope here >> >>>> together to find a solution. >> >>>> Here is the video: http://youtu.be/_TKzRluzg38 >> >>>> >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> Am 25.08.2014 um 08:52 schrieb Jinhang Choi <[email protected]>: >> >>>> >> >>>> Dear Christian, >> >>>> >> >>>> worker log indicates that "GC overhead limit exceeded." >> >>>> >> >>>> would you mind to extend worker's heap memory size at tajo-env.sh? >> >>>> >> >>>> (please refer to >> >>>> >> >>>> http://tajo.apache.org/docs/current/configuration/worker_configuration.html) >> >>>> >> >>>> Sincerely, >> >>>> ---------------------------------------------- >> >>>> Jinhang Choi, CTO. >> >>>> Linewalks Inc. Seoul 137-860, Korea >> >>>> Office: +82 70 7702 3043 >> >>>> FAX: +82 2 2055 0612 >> >>>> >> >>>> -----Original Message----- >> >>>> From: "Christian Schwabe"<[email protected]> >> >>>> To: <[email protected]>; >> >>>> Cc: >> >>>> Sent: 2014-08-25 (월) 15:33:15 >> >>>> Subject: Re: Big big query >> >>>> >> >>>> >> >>>> Hello Hyunsik, >> >>>> >> >>>> sorry. My fault. I will send you another email with the attached >> >>>> logs. >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> Am 25.08.2014 08:28:17, schrieb Hyunsik Choi: >> >>>> >> >>>> Hi Chris, >> >>>> >> >>>> As Jihoon mentioned, it would be better to find the problems if you >> >>>> attach master and worker logs. >> >>>> >> >>>> Thanks, >> >>>> hyunsik >> >>>> >> >>>> >> >>>> On Sun, Aug 24, 2014 at 4:17 PM, Christian Schwabe >> >>>> <[email protected]> wrote: >> >>>> >> >>>> Hello guys >> >>>> >> >>>> i started following query last night and this morning have seen that >> >>>> still ran the query with the fact that has the display of procent not >> >>>> changed and only ran on time. So I have to start again this morning >> >>>> reproduce the query to the error. The result you see in the table >> >>>> below: >> >>>> >> >>>> Running Queries >> >>>> >> >>>> QueryIdQuery Master Started ProgressTimeStatus sql Kill Query >> >>>> q_1408862421753_0001 christians-mbp.fritz.box2014-08-24 08:46:33 45% >> >>>> 15 >> >>>> mins, 48 secQUERY_RUNNINGINSERT INTO TMP_DFKKKO_DFKKOP select >> >>>> pos.validthru, >> >>>> pos.mandt, pos.opbel, pos.opupk, pos.opupz, pos.bukrs, pos.augrs, >> >>>> pos.abwkt, >> >>>> pos.hvorg, pos.tvorg, pos.kofiz, pos.hkont, pos.mwskz, pos.mwszkz, >> >>>> pos.xanza, pos.stakz, pos.astkz, pos.opsta, pos.infoz, pos.inkps, >> >>>> pos.betrh, >> >>>> pos.studt, ko.fikey, ko.blart, ko.herkf, ko.stbel, ko.storb, >> >>>> ko.ernam, >> >>>> ko.cpudt, ko.cputm, ko.bldat, ko.budat from dfkkop_hist pos left join >> >>>> dfkkopw_hist wdh on ( pos.validthru = wdh.validthru and pos.mandt = >> >>>> wdh.mandt and pos.opbel = wdh.opbel and pos.whgrp = wdh.whgrp ) inner >> >>>> join >> >>>> dfkkko_hist ko on ( pos.validthru = ko.validthru and pos.mandt = >> >>>> ko.mandt >> >>>> and pos.opbel = ko.opbel ) >> >>>> >> >>>> Second Screenshot: >> >>>> >> >>>> Running Queries >> >>>> >> >>>> QueryIdQuery Master Started ProgressTimeStatus sql Kill Query >> >>>> q_1408862421753_0001 christians-mbp.fritz.box2014-08-24 08:46:33 43% >> >>>> 23 >> >>>> mins, 21 secQUERY_RUNNINGINSERT INTO TMP_DFKKKO_DFKKOP select >> >>>> pos.validthru, >> >>>> pos.mandt, pos.opbel, pos.opupk, pos.opupz, pos.bukrs, pos.augrs, >> >>>> pos.abwkt, >> >>>> pos.hvorg, pos.tvorg, pos.kofiz, pos.hkont, pos.mwskz, pos.mwszkz, >> >>>> pos.xanza, pos.stakz, pos.astkz, pos.opsta, pos.infoz, pos.inkps, >> >>>> pos.betrh, >> >>>> pos.studt, ko.fikey, ko.blart, ko.herkf, ko.stbel, ko.storb, >> >>>> ko.ernam, >> >>>> ko.cpudt, ko.cputm, ko.bldat, ko.budat from dfkkop_hist pos left join >> >>>> dfkkopw_hist wdh on ( pos.validthru = wdh.validthru and pos.mandt = >> >>>> wdh.mandt and pos.opbel = wdh.opbel and pos.whgrp = wdh.whgrp ) inner >> >>>> join >> >>>> dfkkko_hist ko on ( pos.validthru = ko.validthru and pos.mandt = >> >>>> ko.mandt >> >>>> and pos.opbel = ko.opbel ) >> >>>> >> >>>> Third Screenshot: >> >>>> >> >>>> >> >>>> Finished Queries >> >>>> >> >>>> QueryIdQuery Master Started FinishedTimeStatus sql >> >>>> q_1408862421753_0001christians-mbp.fritz.box 2014-08-24 08:46:33-- >> >>>> QUERY_RUNNINGINSERT INTO TMP_DFKKKO_DFKKOP select pos.validthru, >> >>>> pos.mandt, >> >>>> pos.opbel, pos.opupk, pos.opupz, pos.bukrs, pos.augrs, pos.abwkt, >> >>>> pos.hvorg, >> >>>> pos.tvorg, pos.kofiz, pos.hkont, pos.mwskz, pos.mwszkz, pos.xanza, >> >>>> pos.stakz, pos.astkz, pos.opsta, pos.infoz, pos.inkps, pos.betrh, >> >>>> pos.studt, >> >>>> ko.fikey, ko.blart, ko.herkf, ko.stbel, ko.storb, ko.ernam, ko.cpudt, >> >>>> ko.cputm, ko.bldat, ko.budat from dfkkop_hist pos left join >> >>>> dfkkopw_hist wdh >> >>>> on ( pos.validthru = wdh.validthru and pos.mandt = wdh.mandt and >> >>>> pos.opbel = >> >>>> wdh.opbel and pos.whgrp = wdh.whgrp ) inner join dfkkko_hist ko on ( >> >>>> pos.validthru = ko.validthru and pos.mandt = ko.mandt and pos.opbel = >> >>>> ko.opbel ) >> >>>> >> >>>> As you can see, the query is still running but there is no image data >> >>>> and >> >>>> progress more. >> >>>> >> >>>> I attached a log in which you can see the output in the console. >> >>>> Striking >> >>>> here is the display of the percentage jumps sometimes and not further >> >>>> altered towards the end and remains constant. >> >>>> The data size of the tables to which I JOINS by lead is for >> >>>> dfkkop_hist >> >>>> 5,83GB, dfkkopw_hist 2,47GB and dfkkko_hist 2.35 GB. As I write this, >> >>>> the >> >>>> description of the query is still running. >> >>>> I know these are large amounts of data, but I would have expected >> >>>> which >> >>>> thus constitutes the colloquial no problem. Can you imagine why it >> >>>> comes >> >>>> here to this problem? >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> Best regards, >> >>>> Chris >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>> >> >> >> >> >> >> >> >
