kecookier commented on issue #5251: URL: https://github.com/apache/incubator-gluten/issues/5251#issuecomment-2060287603
The spill directory of task contains more than `1321410(130W+)` files. ``` I20240415 20:58:58.384371 107493 Task.cpp:1111] All drivers (1) finished for task Gluten_Stage_15_TID_3100 after running for 380445 ms. I20240415 20:58:58.384446 107493 Task.cpp:1795] Terminating task Gluten_Stage_15_TID_3100 with state Finished after running for 380445 ms. I20240415 20:58:58.385166 107493 JniWrapper.cc:513] ColumnarBatchOutIterator_nativeClose begin I20240415 20:58:58.385201 107493 JniWrapper.cc:520] nativeClose: getRuntime. I20240415 20:58:58.385212 107493 WholeStageResultIterator.cc:581] [zhaokuo] ~WholeStageResultIteratorMiddleStage() begin I20240415 20:58:58.385221 107493 WholeStageResultIterator.cc:582] [zhaokuo] streamIds_.clear() I20240415 20:58:58.385228 107493 WholeStageResultIterator.cc:584] [zhaokuo] ~WholeStageResultIteratorMiddleStage() end I20240415 20:58:58.385236 107493 WholeStageResultIterator.cc:99] [zhaokuo] ~WholeStageResultIterator() begin I20240415 20:58:58.385244 107493 WholeStageResultIterator.cc:104] [zhaokuo] omittedNodeIds_.clear() and orderedNodeIds_.clear() I20240415 20:58:58.385252 107493 WholeStageResultIterator.cc:107] [zhaokuo] confMap_.clear() I20240415 20:58:58.385264 107493 WholeStageResultIterator.cc:109] [zhaokuo] veloxPlan_.reset() I20240415 20:58:58.385272 107493 WholeStageResultIterator.cc:111] [zhaokuo] task_.reset() I20240415 20:58:58.385284 107493 Task.cpp:402] Gluten_Stage_15_TID_3100[zhaokuo] removeSpillDirectoryIfExists begin I20240415 20:58:58.385293 107493 Task.cpp:411] Gluten_Stage_15_TID_3100[zhaokuo] debugListDirectoryContents begin I20240415 20:58:58.385447 107493 Task.cpp:384] [zhaokuo] first:1 spill file:"/data18/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-27685502-d620-4d84-a59b-f540f4bdbf3f/gluten-spill/e9a92f19-7f07-4d69-b059-84ec2194993f/0_0_1-spill-0-0-0" size:390481968 I20240415 20:58:58.385521 107493 Task.cpp:384] [zhaokuo] first:2 spill file:"/data18/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-27685502-d620-4d84-a59b-f540f4bdbf3f/gluten-spill/e9a92f19-7f07-4d69-b059-84ec2194993f/0_0_1-spill-0-1-1" size:146028957 I20240415 20:58:58.385558 107493 Task.cpp:392] [zhaokuo] /data18/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-27685502-d620-4d84-a59b-f540f4bdbf3f/gluten-spill/e9a92f19-7f07-4d69-b059-84ec2194993f contain: 2 files, total size:536510925 I20240415 20:58:58.385568 107493 Task.cpp:413] [zhaokuo] debugListDirectoryContents end I20240415 20:58:58.385581 107493 Task.cpp:418] [zhaokuo] rmdir begin I20240415 20:58:58.553256 107493 Task.cpp:420] [zhaokuo] rmdir end I20240415 20:58:58.553313 107493 Task.cpp:425] Gluten_Stage_15_TID_3100[zhaokuo] removeSpillDirectoryIfExists end I20240415 20:58:58.553702 107493 WholeStageResultIterator.cc:113] [zhaokuo] ~WholeStageResultIterator() end I20240415 20:58:58.553728 107493 JniWrapper.cc:524] ColumnarBatchOutIterator_nativeClose end. elapsed: 168530887 I20240415 20:58:58.553958 107493 JniWrapper.cc:579] NativeColumnarToRowJniWrapper_nativeClose begin I20240415 20:58:58.553982 107493 JniWrapper.cc:588] NativeColumnarToRowJniWrapper_nativeClose end. elapsed: 17149 I20240415 20:58:58.571107 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[29,0] FILES:70075 SIZE:733.51MB], memory pool: op.2.1.0.HashBuild I20240415 20:59:07.784687 107493 WholeStageResultIterator.cc:184] Spill[WholeStageIterator_root/WholeStageIterator_root]: Trying to request spilling for 8598322 bytes... I20240415 20:59:43.000511 107493 WholeStageResultIterator.cc:197] Spill[WholeStageIterator_root/WholeStageIterator_root]: Successfully spilled out 1391460352 bytes. I20240415 20:59:49.668623 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,0] FILES:64523 SIZE:189.76MB], memory pool: op.2.1.0.HashBuild I20240415 21:00:41.684048 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,1] FILES:65304 SIZE:191.66MB], memory pool: op.2.1.0.HashBuild I20240415 21:01:23.599437 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,2] FILES:65254 SIZE:191.59MB], memory pool: op.2.1.0.HashBuild I20240415 21:02:05.682976 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,3] FILES:65272 SIZE:191.62MB], memory pool: op.2.1.0.HashBuild I20240415 21:02:45.709901 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[29,1] FILES:70013 SIZE:733.07MB], memory pool: op.2.1.0.HashBuild I20240415 21:02:53.737296 107493 WholeStageResultIterator.cc:184] Spill[WholeStageIterator_root/WholeStageIterator_root]: Trying to request spilling for 37119589 bytes... I20240415 21:03:43.228176 107493 WholeStageResultIterator.cc:197] Spill[WholeStageIterator_root/WholeStageIterator_root]: Successfully spilled out 1391460352 bytes. I20240415 21:03:49.602098 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,0] FILES:65312 SIZE:191.65MB], memory pool: op.2.1.0.HashBuild I20240415 21:04:32.030696 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,1] FILES:65005 SIZE:190.83MB], memory pool: op.2.1.0.HashBuild I20240415 21:05:14.320813 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,2] FILES:65242 SIZE:191.39MB], memory pool: op.2.1.0.HashBuild I20240415 21:05:56.492584 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,3] FILES:64829 SIZE:190.37MB], memory pool: op.2.1.0.HashBuild I20240415 21:06:37.040735 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[29,2] FILES:69968 SIZE:732.55MB], memory pool: op.2.1.0.HashBuild I20240415 21:07:16.739650 107493 WholeStageResultIterator.cc:184] Spill[WholeStageIterator_root/WholeStageIterator_root]: Trying to request spilling for 8388608 bytes... I20240415 21:08:22.343214 107493 WholeStageResultIterator.cc:197] Spill[WholeStageIterator_root/WholeStageIterator_root]: Successfully spilled out 1391460352 bytes. I20240415 21:08:36.314782 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,0] FILES:65012 SIZE:190.76MB], memory pool: op.2.1.0.HashBuild I20240415 21:09:19.179255 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,1] FILES:64843 SIZE:190.39MB], memory pool: op.2.1.0.HashBuild I20240415 21:10:01.627730 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,2] FILES:65043 SIZE:190.91MB], memory pool: op.2.1.0.HashBuild I20240415 21:10:41.826995 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,3] FILES:65318 SIZE:191.64MB], memory pool: op.2.1.0.HashBuild I20240415 21:11:23.948865 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[29,3] FILES:70171 SIZE:734.22MB], memory pool: op.2.1.0.HashBuild I20240415 21:12:02.216460 107493 WholeStageResultIterator.cc:184] Spill[WholeStageIterator_root/WholeStageIterator_root]: Trying to request spilling for 8388608 bytes... I20240415 21:13:34.576531 107493 WholeStageResultIterator.cc:197] Spill[WholeStageIterator_root/WholeStageIterator_root]: Successfully spilled out 1391460352 bytes. I20240415 21:13:40.917105 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,0] FILES:65100 SIZE:191.53MB], memory pool: op.2.1.0.HashBuild I20240415 21:14:17.474700 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,1] FILES:65078 SIZE:191.36MB], memory pool: op.2.1.0.HashBuild I20240415 21:14:51.602272 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,2] FILES:64912 SIZE:190.98MB], memory pool: op.2.1.0.HashBuild I20240415 21:15:24.057471 107493 HashBuild.cpp:211] Setup reader to read spilled input from SPILLED PARTITION[ID:[31,3] FILES:65080 SIZE:191.48MB], memory pool: op.2.1.0.HashBuild I20240415 21:16:00.732949 107493 Task.cpp:1111] All drivers (2) finished for task Gluten_Stage_15_TID_3100 after running for 1214475 ms. I20240415 21:16:00.733013 107493 Task.cpp:1795] Terminating task Gluten_Stage_15_TID_3100 with state Finished after running for 1214476 ms. I20240415 21:16:00.872574 107493 JniWrapper.cc:513] ColumnarBatchOutIterator_nativeClose begin I20240415 21:16:00.872622 107493 JniWrapper.cc:520] nativeClose: getRuntime. I20240415 21:16:00.872628 107493 WholeStageResultIterator.cc:581] [zhaokuo] ~WholeStageResultIteratorMiddleStage() begin I20240415 21:16:00.872632 107493 WholeStageResultIterator.cc:582] [zhaokuo] streamIds_.clear() I20240415 21:16:00.872637 107493 WholeStageResultIterator.cc:584] [zhaokuo] ~WholeStageResultIteratorMiddleStage() end I20240415 21:16:00.872640 107493 WholeStageResultIterator.cc:99] [zhaokuo] ~WholeStageResultIterator() begin I20240415 21:16:00.872644 107493 WholeStageResultIterator.cc:104] [zhaokuo] omittedNodeIds_.clear() and orderedNodeIds_.clear() I20240415 21:16:00.872648 107493 WholeStageResultIterator.cc:107] [zhaokuo] confMap_.clear() I20240415 21:16:00.872651 107493 WholeStageResultIterator.cc:109] [zhaokuo] veloxPlan_.reset() I20240415 21:16:00.872654 107493 WholeStageResultIterator.cc:111] [zhaokuo] task_.reset() I20240415 21:16:00.872661 107493 Task.cpp:402] Gluten_Stage_15_TID_3100[zhaokuo] removeSpillDirectoryIfExists begin I20240415 21:16:00.872664 107493 Task.cpp:411] Gluten_Stage_15_TID_3100[zhaokuo] debugListDirectoryContents begin I20240415 21:16:00.873114 107493 Task.cpp:384] [zhaokuo] first:1 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-2-36884-707811" size:2355 I20240415 21:16:00.873181 107493 Task.cpp:384] [zhaokuo] first:2 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-6630-547244" size:2352 I20240415 21:16:00.873217 107493 Task.cpp:384] [zhaokuo] first:3 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-5445-285698" size:2359 I20240415 21:16:00.873234 107493 Task.cpp:384] [zhaokuo] first:4 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-3-2821-738988" size:2345 I20240415 21:16:00.873260 107493 Task.cpp:384] [zhaokuo] first:5 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-2-44029-1235437" size:2359 I20240415 21:16:00.873283 107493 Task.cpp:384] [zhaokuo] first:6 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-56936-337189" size:2371 I20240415 21:16:00.873324 107493 Task.cpp:384] [zhaokuo] first:7 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-1-66449-136499" size:2350 I20240415 21:16:00.873350 107493 Task.cpp:384] [zhaokuo] first:8 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-3-11842-487170" size:2316 I20240415 21:16:00.873368 107493 Task.cpp:384] [zhaokuo] first:9 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-52921-1114155" size:2382 I20240415 21:16:00.873385 107493 Task.cpp:384] [zhaokuo] first:10 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-55527-596141" size:2364 I20240415 21:16:00.873404 107493 Task.cpp:384] [zhaokuo] first:11 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-3-38360-774527" size:2307 I20240415 21:16:00.873422 107493 Task.cpp:384] [zhaokuo] first:12 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-10184-550798" size:2356 I20240415 21:16:00.873446 107493 Task.cpp:384] [zhaokuo] first:13 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-2-7497-678424" size:2361 I20240415 21:16:00.873483 107493 Task.cpp:384] [zhaokuo] first:14 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-39701-39703" size:2377 I20240415 21:16:00.873512 107493 Task.cpp:384] [zhaokuo] first:15 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-1-3136-869156" size:2359 I20240415 21:16:00.873534 107493 Task.cpp:384] [zhaokuo] first:16 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-1-19210-885230" size:2356 I20240415 21:16:00.873566 107493 Task.cpp:384] [zhaokuo] first:17 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-0-9563-1070797" size:2342 I20240415 21:16:00.873600 107493 Task.cpp:384] [zhaokuo] first:18 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-2-63501-473577" size:2354 I20240415 21:16:00.873621 107493 Task.cpp:384] [zhaokuo] first:19 spill file:"/data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694/1_0_1-spill-3-20242-1016144" size:2333 I20240415 21:16:14.085613 107493 Task.cpp:392] [zhaokuo] /data10/hadoop/yarn/nm-local-dir/usercache/hadoop-ba-dealrank/appcache/application_1709200320182_30253983/gluten-1efd6f96-ee9e-49b1-85c5-516014450d5c/gluten-spill/1b5967e6-062f-442d-97b3-2db271f0b694 contain: 1321410 files, total size:7171745932 I20240415 21:16:14.085659 107493 Task.cpp:413] [zhaokuo] debugListDirectoryContents end I20240415 21:16:14.085666 107493 Task.cpp:418] [zhaokuo] rmdir begin I20240415 21:42:23.327833 107493 Task.cpp:420] [zhaokuo] rmdir end I20240415 21:42:23.327896 107493 Task.cpp:425] Gluten_Stage_15_TID_3100[zhaokuo] removeSpillDirectoryIfExists end I20240415 21:42:23.328025 107493 WholeStageResultIterator.cc:113] [zhaokuo] ~WholeStageResultIterator() end I20240415 21:42:23.328063 107493 JniWrapper.cc:524] ColumnarBatchOutIterator_nativeClose end. elapsed: 1582455440727 I20240415 21:42:23.328155 107493 JniWrapper.cc:579] NativeColumnarToRowJniWrapper_nativeClose begin I20240415 21:42:23.328215 107493 JniWrapper.cc:588] NativeColumnarToRowJniWrapper_nativeClose end. elapsed: 54908 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
