Re: generateTreeString causes huge performance problems on dataframe persistence

2015-06-17 Thread Jan-Paul Bultmann
- > From: Jan-Paul Bultmann [mailto:janpaulbultm...@me.com] > Sent: Wednesday, June 17, 2015 6:17 PM > To: User > Subject: generateTreeString causes huge performance problems on dataframe > persistence > > Hey, > I noticed that my code spends hours with `generateTre

RE: generateTreeString causes huge performance problems on dataframe persistence

2015-06-17 Thread Cheng, Hao
to:janpaulbultm...@me.com] Sent: Wednesday, June 17, 2015 6:17 PM To: User Subject: generateTreeString causes huge performance problems on dataframe persistence Hey, I noticed that my code spends hours with `generateTreeString` even though the actual dag/dataframe execution takes seconds. I’m runni

generateTreeString causes huge performance problems on dataframe persistence

2015-06-17 Thread Jan-Paul Bultmann
Hey, I noticed that my code spends hours with `generateTreeString` even though the actual dag/dataframe execution takes seconds. I’m running a query that grows exponential in the number of iterations when evaluated without caching, but should be linear when caching previous results. E.g. r