Hi Mehran What does the full plan look like?You seem to be missing the SCAN operator. Can you post the plans for 1.13 as well? It isn't clear from the table if there is indeed a problem, since you've not shared the query that created this plan.
Also, there have been changes gone in, such as batch sizing, but I'm not sure if the writer would be affected by it. ~ Kunal On 5/4/2019 6:59:19 AM, Mehran.D [BR-PD] <[email protected]> wrote: Dears. I have some jsons to be converted to parquet files. After versin 1.13 I have a very serious issue in this writing. A sample of 100000 record json document takes 5 minutes to finish. I’v included the plan. Overview [http://10.233.50.111:8047/profiles/23329968-0346-e658-b1c9-92fd7dc60d2a#operator-overview] Operator ID Type Avg Setup Time Max Setup Time Avg Process Time Max Process Time Min Wait Time Avg Wait Time Max Wait Time % Fragment Time % Query Time Rows Avg Peak Memory Max Peak Memory 00-xx-00 SCREEN 0.000s 0.000s 1.262s 2.510s 0.004s 0.075s 0.145s 0.82% 0.82% 111,491 10MB 20MB 00-xx-01 PROJECT 0.002s 0.002s 0.001s 0.001s 0.000s 0.000s 0.000s 0.00% 0.00% 1 - - 00-xx-02 PARQUET_WRITER 0.293s 0.293s 50.750s 50.750s 0.000s 0.000s 0.000s 16.44% 16.44% 111,490 - - 00-xx-03 PROJECT_ALLOW_DUP 0.032s 0.032s 2m0s 2m0s 0.000s 0.000s 0.000s 39.01% 39.01% 111,490 13MB 13MB 00-xx-04 PROJECT 16.092s 16.092s 2m15s 2m15s 0.000s 0.000s 0.000s 43.73% 43.73% 111,490 13MB 13MB I do not know what the mothods ‘PROJECT_ALLOW_DUP’ and ‘PROJECT’ does. Please tell me what has changed from 1.13 up to now. I have similar problem in 1.16 release. and Best Regards, [LOGO1] Mehran Dashti Product Leader 09125902452
