[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-27 Thread Weston Pace (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529181#comment-17529181 ] Weston Pace commented on ARROW-16320: - The writing behavior you described seemed odd so I modified

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-27 Thread Zsolt Kegyes-Brassai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528712#comment-17528712 ] Zsolt Kegyes-Brassai commented on ARROW-16320: -- And here is the result of the reading back

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-27 Thread Zsolt Kegyes-Brassai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528661#comment-17528661 ] Zsolt Kegyes-Brassai commented on ARROW-16320: -- Hi [~westonpace]  I tried to create a

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-27 Thread Zsolt Kegyes-Brassai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528616#comment-17528616 ] Zsolt Kegyes-Brassai commented on ARROW-16320: -- It's an error message: {code:java}

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-26 Thread Weston Pace (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528511#comment-17528511 ] Weston Pace commented on ARROW-16320: - What do you get from the following? {noformat} a =

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-26 Thread Zsolt Kegyes-Brassai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527930#comment-17527930 ] Zsolt Kegyes-Brassai commented on ARROW-16320: -- I tried to find what is causing this issue.

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-25 Thread Zsolt Kegyes-Brassai (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527717#comment-17527717 ] Zsolt Kegyes-Brassai commented on ARROW-16320: -- Hi [~westonpace]. Thank you for your prompt

[jira] [Commented] (ARROW-16320) Dataset re-partitioning consumes considerable amount of memory

2022-04-25 Thread Weston Pace (Jira)
[ https://issues.apache.org/jira/browse/ARROW-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527653#comment-17527653 ] Weston Pace commented on ARROW-16320: - How are you measuring memory used? There is a known issue