[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18459
  
great work!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Thanks @holdenk!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged to master. Thanks everyone (especially @shaneknapp & @BryanCutler ) 
:) If anyone sees anything come up in the builds we will revert, but I think 
the multiple runes @shaneknapp's verification means everything is looking good 
:)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
I think we are indeed good to go. I'll merge this back in if no one objects 
before 4pm pacific today.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
That's great to hear @shaneknapp , thanks for all your help getting this 
going!

@cloud-fan , @holdenk since the environment upgrades this has passed tests 
4 time in a row, and I had verified earlier that ArrowTests were being run.  
The worker upgrades appear to be stable and not causing any failures.  Do you 
think this is ok to be merged back in?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79468/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79468 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79468/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
ok, i feel confident that this PR should be g2g:
  - i checked the workers that this PRs builds were on and they didn't 
leave any stray lockfiles
  - i checked ALL workers for stray lockfiles, and only found one from a 
month ago (which i cleaned up)
  - no other spark PRB builds are failing post-upgrade w/system-level issues

so:  +1 from me for merging!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79468 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79468/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79427/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79427 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79427/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79427 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79427/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79363/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79363 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79363/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79363 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79363/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79355/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79355 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79355/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
I haven't seen lock contention before setting up conda enviroments, if it 
happens again lets dig deeper but if its just a one off I wouldn't be too 
worried.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79355 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79355/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
i'd kick off a couple #tbh  :)

On Fri, Jul 7, 2017 at 5:02 PM, Bryan Cutler 
wrote:

> Ok, no prob. I'll kick off another test, maybe that was just a fluke.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Ok, no prob.  I'll kick off another test, maybe that was just a fluke.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
hmm.  i have a feeling w/o looking at the test code that we're creating 
lots of envs, installing things, and then moving on to a new env...  which is 
leading to a race condition w/lockfiles.

i just did a `conda clean --lock` on all of the workers, but i don't think 
that'll fix things long term.

another problem is that i'm heading out of town for the weekend, and won't 
be able to take a deeper look until sunday night at the earliest.  :\



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
@shaneknapp this passed the ArrowTests, but looks like it failed while 
setting up conda for pip-tests because it couldn't acquire a lock
```
Exceeded max retries, giving upError: LOCKERROR: It looks like conda is 
already doing something.
The lock 
[u'/home/sparkivy/per-executor-caches/9/.conda/envs/.pkgs/.conda_lock-80213'] 
was found. Wait for it to finish before continuing.
If you are sure that conda is not running, remove it and try again.
You can also use: $ conda clean --lock
```
Is that the problem you were referring to above?  cc @holdenk 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79346/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79346 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79346/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).
 * This patch **fails PySpark pip packaging tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
one quick comment...  i see that these tests are using the default ivy 
cache of `/home/jenkins/.ivy2/cache`, which is dangerous as other builds and 
whatnot can pollute this w/jars and cause test failures.

what @JoshRosen and i have set up is a per-executor ivy cache for PRB 
builds in `/home/sparkivy/per-executor-caches` on each worker, w/a subdir for 
each jenkins worker's executor (1-12).  the setup code for this can be seen in 
the spark PRB config:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/configure

if you ( @BryanCutler or @wesm ) think this will be a factor in these tests 
(which i feel they could be), hit me up via the contact info in the amplab 
jenkins wiki and i can set you up w/access to see the PRB config and get access 
to the workers if you need it.

https://amplab.cs.berkeley.edu/wiki/jenkins



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79346 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79346/testReport)**
 for PR 18459 at commit 
[`26dfc82`](https://github.com/apache/spark/commit/26dfc82896c3d2a5178b34e46162b3cc06ea8f2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
ArrowTests are verified to be running after forcing this failure:
```
==
FAIL: test_filtered_frame (pyspark.sql.tests.ArrowTests)
--
Traceback (most recent call last):
  File 
"/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests.py", 
line 2698, in test_filtered_frame
self.assertTrue(False)
AssertionError: False is not true
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79339/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79339 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79339/testReport)**
 for PR 18459 at commit 
[`930d624`](https://github.com/apache/spark/commit/930d62456686a5c4d3b69c04c4c7e8ef78674286).
 * This patch **fails PySpark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Great, thanks @shaneknapp!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #79339 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79339/testReport)**
 for PR 18459 at commit 
[`930d624`](https://github.com/apache/spark/commit/930d62456686a5c4d3b69c04c4c7e8ef78674286).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
```
(py3k) [root@amp-jenkins-worker-01 ~]# pip show pandas numpy
Name: pandas
Version: 0.19.2
Summary: Powerful data structures for data analysis, time series,and 
statistics
Home-page: http://pandas.pydata.org
Author: The PyData Development Team
Author-email: pyd...@googlegroups.com
License: BSD
Location: /home/anaconda/envs/py3k/lib/python3.4/site-packages
Requires: python-dateutil, pytz, numpy
---
Name: numpy
Version: 1.11.3
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: NumPy Developers
Author-email: numpy-discuss...@scipy.org
License: BSD
Location: /home/anaconda/envs/py3k/lib/python3.4/site-packages
Requires:
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
this has been done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
ok, i've backed up the existing anaconda installation on all of the 
workers, so i can do the system-wide upgrade and back out if necessary.

i'll post to the dev@ list about upgrading everything and see if we get any 
comments or feedback.

have i mentioned that i can't wait for containerized tests?  :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
alright, this is what a `conda install numpy` would update on the workers:
```
package|build
---|-
blz-removed|0  291 B
dynd-python-removed|0  297 B
libgfortran-3.0.0  |1 281 KB
libiconv-1.14  |0 2.0 MB
mkl-2017.0.3   |0   129.5 MB
hdf5-1.8.17|2 2.0 MB
libpng-1.6.27  |0 219 KB
anaconda-custom|   py34_0   3 KB
freetype-2.5.5 |2 2.5 MB
llvmlite-0.15.0|   py34_0 7.6 MB
numpy-1.11.3   |   py34_0 6.7 MB
abstract-rendering-0.5.1   |  np111py34_0  72 KB
astropy-1.3|  np111py34_0 9.2 MB
bcolz-1.0.0|   py34_0 990 KB
cycler-0.10.0  |   py34_0  11 KB
fontconfig-2.12.1  |3 429 KB
h5py-2.6.0 |  np111py34_2 2.7 MB
nltk-3.2.2 |   py34_0 1.9 MB
numba-0.30.1   |  np111py34_0 2.2 MB
numexpr-2.6.1  |  np111py34_2 354 KB
patsy-0.4.1|   py34_0 372 KB
scipy-0.18.1   |  np111py34_131.3 MB
datashape-0.5.4|   py34_0  99 KB
pandas-0.19.2  |  np111py34_117.1 MB
pytables-3.3.0 |  np111py34_0 5.8 MB
scikit-learn-0.18.1|  np111py34_111.3 MB
bokeh-0.10.0   |   py34_0 3.9 MB
odo-0.4.2  |   py34_0 177 KB
statsmodels-0.6.1  |  np111py34_1 5.2 MB
blaze-core-0.9.0   |   py34_0 541 KB
matplotlib-1.5.1   |  np111py34_0 8.5 MB
scikit-image-0.12.3|  np111py34_126.7 MB

   Total:   279.5 MB
```

important upgrades happening in there:
  * numpy to 1.11.3
  * pandas to 0.19.2

@JoshRosen : i can update amp-jenkins-worker-01 if you'd like to test 
outside of the regular PRB workers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
No problem, I used to have a life too ;)
I need to push an update to fix the conflict, but I'll hold off until we
are ready to run again.  Thanks!

On Wed, Jul 5, 2017 at 5:17 PM, shane  wrote:

> no, i will tomorrow though. real life[tm] took over my day today.
>
> On Wed, Jul 5, 2017 at 4:54 PM, Bryan Cutler 
> wrote:
>
> > Hi @shaneknapp , did you happen to get a
> > chance to work on this yet?
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > , or
> mute
> > the thread
> >  auth/ABiDrB-SIVg6u0lTuq_Lx-oGTx-1XLNQks5sLCJJgaJpZM4OITbk>
>
> > .
> >
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
no, i will tomorrow though.  real life[tm] took over my day today.

On Wed, Jul 5, 2017 at 4:54 PM, Bryan Cutler 
wrote:

> Hi @shaneknapp , did you happen to get a
> chance to work on this yet?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Hi @shaneknapp, did you happen to get a chance to work on this yet?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
nah, i got it.

On Sat, Jul 1, 2017 at 7:19 PM, Holden Karau 
wrote:

> @shaneknapp  - I can do the update if you
> want next week? Let me know :)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
@shaneknapp - I can do the update if you want next week? Let me know :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Great, thanks @shaneknapp! Just let me know what I can do to help

On Jun 30, 2017 4:52 PM, "shane"  wrote:

> i won't have time to think about and do something until monday... but i
> have some ideas.
>
> On Fri, Jun 30, 2017 at 4:29 PM, Bryan Cutler 
> wrote:
>
> > Thanks for checking on that Wes! @shaneknapp
> >  and @holdenk  >
> > I definitely don't want you to go through dependency hell... 👎 I'm 
not
> > too sure how to resolve this since I have no access to Jenkins. At this
> > point would it be better to try setting these tests up through a virtual
> > env again?
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > , or
> mute
> > the thread
> >  jJhXfFO6BNh8ZAmIxtcWks5sJYT3gaJpZM4OITbk>
> > .
> >
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
i won't have time to think about and do something until monday...  but i
have some ideas.

On Fri, Jun 30, 2017 at 4:29 PM, Bryan Cutler 
wrote:

> Thanks for checking on that Wes! @shaneknapp
>  and @holdenk 
> I definitely don't want you to go through dependency hell... 👎 I'm not
> too sure how to resolve this since I have no access to Jenkins. At this
> point would it be better to try setting these tests up through a virtual
> env again?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Thanks for checking on that Wes!  @shaneknapp and @holdenk I definitely 
don't want you to go through dependency hell... :-1:   I'm not too sure how to 
resolve this since I have no access to Jenkins.  At this point would it be 
better to try setting these tests up through a virtual env again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[wesm]

Github user wesm commented on the issue:

https://github.com/apache/spark/pull/18459
  
I just checked and pandas 0.18.1 is the minimum supported version. It's 
possible with some effort we could maintain against 0.17.x (which is the first 
release to have datetime with TZ column types) but it would have to wait for 
the Arrow 0.5.0 release. Since pandas doesn't do maintenance releases for major 
versions I am not sure if it's worth the trouble. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Hmm, that's a pretty old version of Pandas also.  I'm not too sure if that 
would pose a problem too. @wesm, do you have any advice on compatibility for 
manually setting up the Spark Jenkins workers with pyarrow 0.4.1 that currently 
have numpy 1.9.2 and pandas 0.15.2?  At this point, I think only the minimum 
upgrades to allow the pyspark arrow tests to pass want to be done.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
pandas is currently at 0.15.2.  

and no, no plans to upgrade unless it's forced by dependency hell.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Hi @shaneknapp, I should have probably asked what version of Pandas is 
currently installed and if you will be upgrading that also with numpy?  Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
It looks like the minimum version in the pyarrow conda recipie is 1.11.x 
[link](https://github.com/conda-forge/pyarrow-feedstock/blob/master/appveyor.yml#L13).
  Up to version 1.13.x is supported, so I would think the most recent is 
preferable unless that is too big of jump.  What do you suggest @holdenk, are 
the any problems you know of using 1.13?  I would have thought installing 
pyarrow would have upgraded numpy also right? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/18459
  
is there a specific version that you would like installed (current is 
1.9.2)?  

i'll do the this as i need to update some setup scripts and whatnot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Thanks @holdenk , I thought there was probably some kind of import error 
causing the tests to be skipped.  I'll see if I can update the min numpy ver.  
I'm not sure its such a good idea to skip the tests on import error.  For a 
users that don't want to install pyarrow, this is good so that they can still 
run tests locally, but it will be hard for automated testing to catch issues 
like this if they crop up in the future.  Any ideas?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
So I went to go take a look on the host and it seems that the numpy version 
is sufficiently out of date it fails.
cc @shaneknapp - would it be ok to install a modern version of numpy in the 
env? (I can do this if you don't mind). Also we should probably update pyarrows 
conda forge recipe to point to a higher minimum version of numpy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
The error is:

> >>> import pyarrow
> RuntimeError: module compiled against API version a but this version of 
numpy is 9
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
"/home/anaconda/envs/py3k/lib/python3.4/site-packages/pyarrow/__init__.py", 
line 28, in 
> from pyarrow.lib import cpu_count, set_cpu_count
>   File "pyarrow/lib.pyx", line 40, in init pyarrow.lib 
(/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:62119)
> ImportError: numpy.core.multiarray failed to import
> >>> 
> 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78842/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78842 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78842/testReport)**
 for PR 18459 at commit 
[`930d624`](https://github.com/apache/spark/commit/930d62456686a5c4d3b69c04c4c7e8ef78674286).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78842 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78842/testReport)**
 for PR 18459 at commit 
[`930d624`](https://github.com/apache/spark/commit/930d62456686a5c4d3b69c04c4c7e8ef78674286).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[holdenk]

Github user holdenk commented on the issue:

https://github.com/apache/spark/pull/18459
  
I think strangely enough it's only persisted on failure. I'd just put an
assert false true at the end to verify the test is running, then take it
out and we can merge it. Sound good?

On Wed, Jun 28, 2017 at 4:58 PM Bryan Cutler 
wrote:

> @holdenk  @HyukjinKwon
>  any ideas what caused this last error?
> It look like pyspark.sql.tests ran fine, which should have run the Arrow
> tests assuming pyarrow was on the worker but Is there some way to check
> /home/jenkins/workspace/SparkPullRequestBuilder/python/unit-tests.log to
> verify this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>
-- 
Cell : 425-233-8271



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18459
  
Thanks for cc'ing me. I am investigating this here - 
https://github.com/apache/spark/pull/18456.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
@holdenk @HyukjinKwon any ideas what caused this last error?
It look like pyspark.sql.tests ran fine, which should have run the Arrow 
tests assuming pyarrow was on the worker but Is there some way to check 
/home/jenkins/workspace/SparkPullRequestBuilder/python/unit-tests.log to verify 
this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78822/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78822 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78822/testReport)**
 for PR 18459 at commit 
[`480849b`](https://github.com/apache/spark/commit/480849bcfa608146edac30397adfa5baf1331bed).
 * This patch **fails due to an unknown error code, -10**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78822 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78822/testReport)**
 for PR 18459 at commit 
[`480849b`](https://github.com/apache/spark/commit/480849bcfa608146edac30397adfa5baf1331bed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18459
  
Jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18459
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78818/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78818 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78818/testReport)**
 for PR 18459 at commit 
[`480849b`](https://github.com/apache/spark/commit/480849bcfa608146edac30397adfa5baf1331bed).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18459: [SPARK-13534][PYSPARK] Using Apache Arrow to increase pe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18459
  
**[Test build #78818 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78818/testReport)**
 for PR 18459 at commit 
[`480849b`](https://github.com/apache/spark/commit/480849bcfa608146edac30397adfa5baf1331bed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org