[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-115032139 All right. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 closed the pull request at: https://github.com/apache/spark/pull/4269 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-114709413 Hi @JoshRosen, no problem. Thank you for considering this PR. I try to revise my code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-113661417 Sorry to confuse you, I agree with you. As a first step, we should rewrite run-tests in Python, then append new features. I took a look at #6866, I think it has some useful functions to rewrite bash code into Python. If you don't mind, I want to wait to merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-5482][PySpark] Allow individual te...
Github user potix2 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4269#discussion_r32794752
--- Diff: python/tests/common ---
@@ -0,0 +1,133 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the License); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+function run_test() {
+echo -en Running test: $1 ... | tee -a $LOG_FILE
+start=$(date +%s)
+SPARK_TESTING=1 time $FWDIR/bin/pyspark $1 $LOG_FILE 21
+
+FAILED=$((PIPESTATUS[0]||$FAILED))
+
+# Fail and exit on the first test failure.
+if [[ $FAILED != 0 ]]; then
+cat $LOG_FILE | grep -v ^[0-9][0-9]* # filter all lines starting
with a number.
+echo -en \033[31m # Red
+echo Had test failures; see logs.
+echo -en \033[0m # No color
+exit -1
+else
+now=$(date +%s)
+echo ok ($(($now - $start))s)
+fi
+}
+
+function run_core_tests() {
+if [ $DO_CORE_TESTS == 0 ]; then
+return 0
+fi
+
+echo Run core tests ...
+run_test pyspark.rdd
+run_test pyspark.context
+run_test pyspark.conf
+run_test pyspark.broadcast
+run_test pyspark.accumulators
+run_test pyspark.serializers
+run_test pyspark.profiler
+run_test pyspark.shuffle
+run_test pyspark.tests
+}
+
+function run_sql_tests() {
+if [ $DO_SQL_TESTS == 0 ]; then
+return 0
+fi
+
+echo Run sql tests ...
+run_test pyspark.sql.types
+run_test pyspark.sql.context
+run_test pyspark.sql.column
+run_test pyspark.sql.dataframe
+run_test pyspark.sql.group
+run_test pyspark.sql.functions
+run_test pyspark.sql.readwriter
+run_test pyspark.sql.window
+run_test pyspark.sql.tests
+}
+
+function run_mllib_tests() {
+if [ $DO_MLLIB_TESTS == 0 ]; then
--- End diff --
I think it is not good, because if we remove these conditions from here, we
must put them into the `tests/default`, `tests/pypy.sh` and another runner
script for a particular python version. @davies pointed out that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-5482][PySpark] Allow individual te...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-113332096 @JoshRosen, I haven't started rewriting in python yet, so it's no problem that I make it to be out of scope for this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-5482][PySpark] Allow individual te...
Github user potix2 commented on a diff in the pull request: https://github.com/apache/spark/pull/4269#discussion_r32791057 --- Diff: python/tests/pypy.sh --- @@ -0,0 +1,33 @@ +# -*- coding: utf-8 -*- --- End diff -- ok, I remove them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-108163829 @JoshRosen Should we define the list of supported python versions in a test script? I found the below versions in a current python/run-tests. * 2.6 * 3.4 * pypy --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-108194195 Thank you, @JoshRosen. I got it. There is no problem to rewrite in Python. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/4269#issuecomment-107980205 ok, I understand it. I try to split run-tests and rebase this branch onto master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Examples] fix deprecated method use in HBaseT...
Github user potix2 closed the pull request at: https://github.com/apache/spark/pull/4725 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Examples] fix deprecated method use in HBaseT...
Github user potix2 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4725#discussion_r25161690
--- Diff: examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala
---
@@ -36,7 +36,7 @@ object HBaseTest {
// Initialize hBase table if necessary
val admin = new HBaseAdmin(conf)
if (!admin.isTableAvailable(args(0))) {
- val tableDesc = new HTableDescriptor(args(0))
+ val tableDesc = new HTableDescriptor(TableName.valueOf(args(0)))
--- End diff --
Sorry, I didn't know when that constructor was added. I understand my
proposal makes the compatibility of the earliest version broken.
The other deprication is nothing in the HBase examples, I close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Examples] fix deprecated method use in HBaseT...
GitHub user potix2 opened a pull request: https://github.com/apache/spark/pull/4725 [Examples] fix deprecated method use in HBaseTest HTableDescriptor(String name) is deprecated. https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/TableName.html You can merge this pull request into a Git repository by running: $ git pull https://github.com/potix2/spark fix-warning-hbase-example Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4725.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4725 commit f613b861afa037f78fc981933789cfc730c9a062 Author: Katsunori Kanda pot...@gmail.com Date: 2015-02-23T10:21:16Z [Examples] fix deprecated method use in HBaseTest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] Update default Spark version to 1.2.1
GitHub user potix2 opened a pull request: https://github.com/apache/spark/pull/4566 [EC2] Update default Spark version to 1.2.1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/potix2/spark ec2-update-version-1-2-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4566.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4566 commit 77e7840d17ed8b9cc050d25256522b9901f48a1e Author: Katsunori Kanda pot...@gmail.com Date: 2015-02-12T12:25:30Z [EC2] Update default Spark version to 1.2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5482][PySpark] Allow individual test su...
GitHub user potix2 opened a pull request: https://github.com/apache/spark/pull/4269 [SPARK-5482][PySpark] Allow individual test suites in python/run-tests Add options to run individual test suites in python/run-tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/potix2/spark spark-5482 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4269.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4269 commit 7ebbb861d4c2231dfb02a1f3b1e16eae61ea1ac4 Author: Katsunori Kanda pot...@gmail.com Date: 2015-01-29T13:38:43Z [SPARK-5482][PySpark] Allow individual test suites in python/run-tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4735]Spark SQL UDF doesn't support 0 ar...
GitHub user potix2 opened a pull request: https://github.com/apache/spark/pull/3604 [SPARK-4735]Spark SQL UDF doesn't support 0 arguments I fixed the udf bug. https://issues.apache.org/jira/browse/SPARK-4735 You can merge this pull request into a Git repository by running: $ git pull https://github.com/potix2/spark bugfix-4735 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3604.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3604 commit 025537a1ec966fa34330fbbc1ab29c2d3d9943cf Author: Katsunori Kanda ka...@amoad.com Date: 2014-12-04T11:52:06Z Add UdfRegistration.registerFunction() for Function0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4735] Spark SQL UDF doesn't support 0 a...
Github user potix2 commented on the pull request: https://github.com/apache/spark/pull/3604#issuecomment-65736186 @chenghao-intel sorry, it's same. I'll close mine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4735] Spark SQL UDF doesn't support 0 a...
Github user potix2 closed the pull request at: https://github.com/apache/spark/pull/3604 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
