[GitHub] spark issue #15011: [SPARK-17122][SQL]support drop current database
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/15011 Thanks a lot @gatorsmile @cloud-fan @adrian-wang ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14169: [SPARK-16515][SQL]set default record reader and writer f...
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/14169 Hi Spark guys, Could you please help to review this PR to merge it in Spark 2.0.0 ? Thanks in advance ! Best Regards, Yi --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14169: [WIP][SPARK-16515][SQL]set default record reader and wri...
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/14169 Hi, Cool ! All of my cases relative to transformation script PASSED after applying this PR . Could Spark guys please review this codes to merge this PR ? Thanks a lots ! Best Regards Yi --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13542: [SPARK-15730][SQL] Respect the --hiveconf in the spark-s...
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/13542 Hi, Could you please help to review this PR to merge it in 2.0.0 ? this broken thing blocked our testing. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13542: [SPARK-15730][SQL] Respect the --hiveconf in the spark-s...
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/13542 Hi Spark community ï¼ Could you please help to review this PR to merge in 2.0.0 ? Since this bug has broken our real cases. Thanks in advance ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13542: [SPARK-15730][SQL][WIP] Respect the --hiveconf in the sp...
Github user jameszhouyi commented on the issue: https://github.com/apache/spark/pull/13542 Hi @chenghao-intel This PR is tested and it's OK for my case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SPARK-11972][SQL]fix commands th...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-161855688 Thanks @marmbrus for your response. This is regression bug(not found in 1.5.X) so hopefully it can be fixed in 1.6.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SPARK-11972][SQL]fix commands th...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-161215023 This is critical bug. Strong hopefully it can be fixed and merged in 1.6.0. Thanks ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SPARK-11972][SQL]fix commands th...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-159505641 Hi @adrian-wang , For SPARK-11972, the case passed now after applying the patch.Thanks ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4226][SQL]Add subquery (not) in/exists ...
Github user jameszhouyi commented on the pull request:
https://github.com/apache/spark/pull/9055#issuecomment-153918005
Thank you @gatorsmile for your suggestion.
I think this feature("IN" sub query) is necessary for Spark SQL engine as
SQL-on-Hadoop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4226][SQL]Add subquery (not) in/exists ...
Github user jameszhouyi commented on the pull request:
https://github.com/apache/spark/pull/9055#issuecomment-152941016
Hi @yhuai ,
This missing feature("IN" sub query) in Spark SQL blocked our real-world
case. Could you please help to review this PR ? Strongly hopefully this PR
feature can be merged in Spark 1.6.0 ( I saw the Hive implementation supported
such feature ). Thanks in advanced !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-142917749 Hi @yhuai , I saw the PR is ready for some time. could you help to review this PR. Hopefully it can be fixed in 1.5.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10310][SQL]Using \t as the field delime...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8476#issuecomment-140675429 I saw the issue marked as 'Target Version 1.5.1' and hopefully it can be merged in 1.5.1... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-138538607 Hi @chenghao-intel , We can pass the case and also get better performance than before optimization for cross join after applying the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-138741073 After optimized patch , we can see "CartesianProduct" optimized to "BroadcastNestedLoopJoin" from physical plan for cross join. The benchmark result showed ~42% performance gain(15m1s vs. 26m37s). == Physical Plan == TungstenProject [concat(cast(s_store_sk#454L as string),_,s_store_name#455) AS store_ID#444,pr_review_date#447,pr_review_content#453] BroadcastNestedLoopJoin BuildRight, Inner, Some((locate(lower(s_store_name#455),lower(pr_review_content#453),1) >= 1)) HiveTableScan [pr_review_date#447,pr_review_content#453], (MetastoreRelation bigbench, product_reviews, Some(pr)) HiveTableScan [s_store_sk#454L,s_store_name#455], (MetastoreRelation bigbench, temp_stores_with_regression, Some(stores_with_regression)) Code Generation: true --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10310][SQL]Using \t as the field delime...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8476#issuecomment-13695 This is real-world case using Spark SQL and hopefully it can be fixed/merged in Spark 1.5.0.Thanks in advance ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10310][SQL]Using \t as the field delime...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8476#issuecomment-136362940 Apply the spark master code(commit 8d2ab75d3b71b632f2394f2453af32f417cb45e5) with this PR patch, the previous broken cases can be passed now.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10130] [SQL] type coercion for IF shoul...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/8331#issuecomment-133317182 This is blocker issue and hopefully it is fixed in Spark 1.5.0, Thanks ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-8064, build against Hive 1.2.1
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/7191#issuecomment-127498023 Hi @steveloughran , Have you came across below errors when build the spark? Please correct me if any missing or wrong. (Build command: mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests clean package) [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 14:37 min [INFO] Finished at: 2015-08-04T14:27:08+08:00 [INFO] Final Memory: 65M/478M [INFO] [ERROR] Failed to execute goal on project spark-hive_2.10: Could not resolve dependencies for project org.apache.spark:spark-hive_2.10:jar:1.5.0-SNAPSHOT: The following artifacts could not be resolved: org.spark-project.hive:hive-exec:jar:1.2.1.spark, org.spark-project.hive:hive-metastore:jar:1.2.1.spark: Failure to find org.spark-project.hive:hive-exec:jar:1.2.1.spark in http://foxcove3.sh.test.com:8081/artifactory/libs-release was cached in the local repository, resolution will not be reattempted until the update interval of libs-release has elapsed or updates are forced - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :spark-hive_2.10 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7119][SQL]Give script a default serde w...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/6638#issuecomment-125098294 Apply this PR based on commit id 'c025c3d0a1fdfbc45b64db9c871176b40b4a7b9b' and the case relative to script transform can pass now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7119][SQL]Give script a default serde w...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/6638#issuecomment-119795927 Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7119][SQL]Give script a default serde w...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/6638#issuecomment-119771895 Hi, I saw the 'Merged build finished. Test FAILed.' if there is a latest version for fix ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7119][SQL] ScriptTransform should also ...
Github user jameszhouyi commented on the pull request:
https://github.com/apache/spark/pull/5688#issuecomment-104464124
@viirya , please see below query details with Using script transform:
ADD FILE ${env:BIG_BENCH_QUERIES_DIR}/Resources/bigbenchqueriesmr.jar;
--CREATE RESULT TABLE. Store query result externally in
output_dir/qXXresult/
DROP TABLE IF EXISTS ${hiveconf:RESULT_TABLE};
CREATE TABLE ${hiveconf:RESULT_TABLE} (
pid1 BIGINT,
pid2 BIGINT,
cnt BIGINT
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS ${env:BIG_BENCH_hive_default_fileformat_result_table} LOCATION
'${hiveconf:RESULT_DIR}';
-- the real query part
--Find the most frequent ones
INSERT INTO TABLE ${hiveconf:RESULT_TABLE}
SELECT pid1, pid2, COUNT (*) AS cnt
FROM (
--Make items basket
FROM (
-- Joining two tables
SELECT s.ss_ticket_number AS oid , s.ss_item_sk AS pid
FROM store_sales s
INNER JOIN item i ON (s.ss_item_sk = i.i_item_sk)
WHERE i.i_category_id in (${hiveconf:q01_i_category_id_IN})
AND s.ss_store_sk in (${hiveconf:q01_ss_store_sk_IN})
CLUSTER BY oid
) q01_map_output
REDUCE q01_map_output.oid, q01_map_output.pid
USING '${env:BIG_BENCH_JAVA} ${env:BIG_BENCH_java_child_process_xmx} -cp
bigbenchqueriesmr.jar de.bankmark.bigbench.queries.q01.Red -ITEM_SET_MAX
${hiveconf:q01_NPATH_ITEM_SET_MAX} '
AS (pid1 BIGINT, pid2 BIGINT)
) q01_temp_basket
GROUP BY pid1, pid2
HAVING COUNT (pid1) ${hiveconf:q01_COUNT_pid1_greater}
CLUSTER BY pid1 ,cnt ,pid2
;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7119][SQL] ScriptTransform should also ...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/5688#issuecomment-103314083 Hi, I tried to apply this patch to run such SQL query with 'Using' Script operation and found below error throwing: 15/04/27 09:46:48 WARN scheduler.TaskSetManager: Lost task 9.0 in stage 132.0 (TID 2859, bignode3): java.lang.ClassCastException: org.apache.spark.sql.types.UTF8String cannot be cast to java.lang.Long at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:110) at org.apache.spark.sql.catalyst.expressions.GenericRow.getLong(rows.scala:88) at org.apache.spark.sql.execution.SparkSqlSerializer2$$anonfun$createSerializationFunction$1.apply(SparkSqlSerializer2.scala:217) at org.apache.spark.sql.execution.SparkSqlSerializer2$$anonfun$createSerializationFunction$1.apply(SparkSqlSerializer2.scala:169) at org.apache.spark.sql.execution.Serializer2SerializationStream.writeObject(SparkSqlSerializer2.scala:58) at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:211) at org.apache.spark.util.collection.ExternalSorter.spillToPartitionFiles(ExternalSorter.scala:390) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:222) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/4035#issuecomment-70643624 Sorry for opened accidentally..please kindly close it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
GitHub user jameszhouyi opened a pull request: https://github.com/apache/spark/pull/4035 Merge pull request #1 from apache/master [SPARK-2140] Updating heap memory calculation for YARN stable and alpha. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jameszhouyi/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4035.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4035 commit 1ff02e202f4527b4a0c639eff5794713644c4b81 Author: Yi Zhou james_zhouyi2...@126.com Date: 2014-09-11T15:05:56Z Merge pull request #1 from apache/master [SPARK-2140] Updating heap memory calculation for YARN stable and alpha. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3786] [PySpark] speedup tests
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2646#issuecomment-58205767 Hi @davies @JoshRosen Found below errors after add 'time' in run-tests Running PySpark tests. Output is in python/unit-tests.log. Testing with Python version: Python 2.6.6 Run core tests ... Running test: pyspark/rdd.py ./python/run-tests: line 37: time: command not found ./python/run-tests: line 37: time: command not found --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3786] [PySpark] speedup tests
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2646#issuecomment-58307284 Hi @davies , The error have been fixed via 'yum install time'. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2444#issuecomment-57071944 Hi @pwendell , After this commit, for spark-perf will complain 'not found slaves' when run ./bin/run... so have to modify from slaves.template to slaves manually ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2359#issuecomment-55368800 The issue found in apache/spark branch-1.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2359#issuecomment-55369150 Hi andrewor14, I also tested this on apache/spark master, there is no errors as like yours. I am not sure why causes this inconsistent behavior on master v.s. branch-1.1. Could you please confirm this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
Github user jameszhouyi commented on the pull request: https://github.com/apache/spark/pull/2359#issuecomment-55473168 Hi @andrewor14 , I just got the the latest branch-1.1 and run again, but still run across the issue like below details, please kindly review below (I ran it on CentOS6.4). $ git status On branch branch-1.1 $ ./dev/run-tests Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Branch 1.1
GitHub user jameszhouyi opened a pull request: https://github.com/apache/spark/pull/2353 Branch 1.1 Symptom: Run ./dev/run-tests and dump outputs as following: SBT_MAVEN_PROFILES_ARGS=-Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl [Warn] Java 8 tests will not run because JDK version is 1.8. = Running Apache RAT checks = RAT checks passed. = Running Scala style checks = Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ Possible Cause: I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like echo -e q\n | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \ scalastyle.txt echo -e q\n | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \ scalastyle.txt From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'.. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/spark branch-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2353 commit ee7d2cc1a935da62de968799c0ecc6f98e43361a Author: Cheng Lian lian.cs@gmail.com Date: 2014-08-14T00:37:55Z [SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian lian.cs@gmail.com Closes #1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer (cherry picked from commit 376a82e196e102ef49b9722e8be0b01ac5890a8b) Signed-off-by: Michael Armbrust mich...@databricks.com commit e8e7f17e1e6d84268421dbfa315850b07a8a4c15 Author: Michael Armbrust mich...@databricks.com Date: 2014-08-14T00:40:59Z [SPARK-2935][SQL]Fix parquet predicate push down bug Author: Michael Armbrust mich...@databricks.com Closes #1863 from marmbrus/parquetPredicates and squashes the following commits: 10ad202 [Michael Armbrust] left = right f249158 [Michael Armbrust] quiet parquet tests. 802da5b [Michael Armbrust] Add test case. eab2eda [Michael Armbrust] Fix parquet predicate push down bug (cherry picked from commit 9fde1ff5fc114b5edb755ed40944607419b62184) Signed-off-by: Michael Armbrust mich...@databricks.com commit b5b632c8cd02fd1e65ebd22216d20ec76715fc5d Author: Kousuke Saruta saru...@oss.nttdata.co.jp Date: 2014-08-14T00:42:38Z [SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #1891 from sarutak/SPARK-2970 and squashes the following commits: 4a2d2fe [Kousuke Saruta] Modified comment style 8bd833c [Kousuke Saruta] Modified style 6c0997c [Kousuke Saruta] Modified the timing of shutdown hook execution. It should be executed before shutdown hook of o.a.h.f.FileSystem (cherry picked from commit 905dc4b405e679feb145f5e6b35e952db2442e0d) Signed-off-by: Michael Armbrust mich...@databricks.com commit a8d2649719b3d8fdb1eed29ef179a6a896b3e37a Author: guowei guo...@upyoo.com Date: 2014-08-14T00:45:24Z [SPARK-2986] [SQL] fixed: setting properties does not effect it seems that set command does not run by SparkSQLDriver. it runs on hive api. user can not change reduce number by setting spark.sql.shuffle.partitions but i think setting hive properties seems just a role to spark sql. Author: guowei guo...@upyoo.com Closes #1904 from guowei2/temp-branch and squashes the following commits: 7d47dde [guowei] fixed: setting properties like spark.sql.shuffle.partitions does not effective (cherry picked from
[GitHub] spark pull request: [SPARK-3480] Throws out Not a valid command 'y...
Github user jameszhouyi closed the pull request at: https://github.com/apache/spark/pull/2353 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3480] Throws out Not a valid command 'y...
GitHub user jameszhouyi reopened a pull request: https://github.com/apache/spark/pull/2353 [SPARK-3480] Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' Symptom: Run ./dev/run-tests and dump outputs as following: SBT_MAVEN_PROFILES_ARGS=-Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl [Warn] Java 8 tests will not run because JDK version is 1.8. Running Apache RAT checks RAT checks passed. Running Scala style checks Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ Possible Cause: I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like echo -e q\n | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \ echo -e q\n | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \ From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'.. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/spark branch-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2353 commit ee7d2cc1a935da62de968799c0ecc6f98e43361a Author: Cheng Lian lian.cs@gmail.com Date: 2014-08-14T00:37:55Z [SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian lian.cs@gmail.com Closes #1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer (cherry picked from commit 376a82e196e102ef49b9722e8be0b01ac5890a8b) Signed-off-by: Michael Armbrust mich...@databricks.com commit e8e7f17e1e6d84268421dbfa315850b07a8a4c15 Author: Michael Armbrust mich...@databricks.com Date: 2014-08-14T00:40:59Z [SPARK-2935][SQL]Fix parquet predicate push down bug Author: Michael Armbrust mich...@databricks.com Closes #1863 from marmbrus/parquetPredicates and squashes the following commits: 10ad202 [Michael Armbrust] left = right f249158 [Michael Armbrust] quiet parquet tests. 802da5b [Michael Armbrust] Add test case. eab2eda [Michael Armbrust] Fix parquet predicate push down bug (cherry picked from commit 9fde1ff5fc114b5edb755ed40944607419b62184) Signed-off-by: Michael Armbrust mich...@databricks.com commit b5b632c8cd02fd1e65ebd22216d20ec76715fc5d Author: Kousuke Saruta saru...@oss.nttdata.co.jp Date: 2014-08-14T00:42:38Z [SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #1891 from sarutak/SPARK-2970 and squashes the following commits: 4a2d2fe [Kousuke Saruta] Modified comment style 8bd833c [Kousuke Saruta] Modified style 6c0997c [Kousuke Saruta] Modified the timing of shutdown hook execution. It should be executed before shutdown hook of o.a.h.f.FileSystem (cherry picked from commit 905dc4b405e679feb145f5e6b35e952db2442e0d) Signed-off-by: Michael Armbrust mich...@databricks.com commit a8d2649719b3d8fdb1eed29ef179a6a896b3e37a Author: guowei guo...@upyoo.com Date: 2014-08-14T00:45:24Z [SPARK-2986] [SQL] fixed: setting properties does not effect it seems that set command does not run by SparkSQLDriver. it runs on hive api. user can not change reduce number by setting spark.sql.shuffle.partitions but i think setting hive properties seems just a role to spark sql. Author: guowei guo...@upyoo.com Closes #1904 from guowei2/temp-branch and squashes the following commits: 7d47dde [guowei] fixed: setting properties like spark.sql.shuffle.partitions does not effective (cherry picked from commit 63d637ca8559d4344d1661500b8ad868bb47) Signed-off-by: Michael Armbrust mich...@databricks.com commit c6cb55a784ba8f9e5c4e7aadcc3ec9dce24f49ee Author: Patrick Wendell pwend
[GitHub] spark pull request: [SPARK-3480] Throws out Not a valid command 'y...
Github user jameszhouyi closed the pull request at: https://github.com/apache/spark/pull/2353 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3480] Throws out Not a valid command 'y...
GitHub user jameszhouyi reopened a pull request: https://github.com/apache/spark/pull/2353 [SPARK-3480] Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' Symptom: Run ./dev/run-tests and dump outputs as following: SBT_MAVEN_PROFILES_ARGS=-Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl [Warn] Java 8 tests will not run because JDK version is 1.8. Running Apache RAT checks RAT checks passed. Running Scala style checks Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ Possible Cause: I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like echo -e q\n | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \ echo -e q\n | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \ From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'.. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/spark branch-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2353.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2353 commit ee7d2cc1a935da62de968799c0ecc6f98e43361a Author: Cheng Lian lian.cs@gmail.com Date: 2014-08-14T00:37:55Z [SPARK-2650][SQL] More precise initial buffer size estimation for in-memory column buffer This is a follow up of #1880. Since the row number within a single batch is known, we can estimate a much more precise initial buffer size when building an in-memory column buffer. Author: Cheng Lian lian.cs@gmail.com Closes #1901 from liancheng/precise-init-buffer-size and squashes the following commits: d5501fa [Cheng Lian] More precise initial buffer size estimation for in-memory column buffer (cherry picked from commit 376a82e196e102ef49b9722e8be0b01ac5890a8b) Signed-off-by: Michael Armbrust mich...@databricks.com commit e8e7f17e1e6d84268421dbfa315850b07a8a4c15 Author: Michael Armbrust mich...@databricks.com Date: 2014-08-14T00:40:59Z [SPARK-2935][SQL]Fix parquet predicate push down bug Author: Michael Armbrust mich...@databricks.com Closes #1863 from marmbrus/parquetPredicates and squashes the following commits: 10ad202 [Michael Armbrust] left = right f249158 [Michael Armbrust] quiet parquet tests. 802da5b [Michael Armbrust] Add test case. eab2eda [Michael Armbrust] Fix parquet predicate push down bug (cherry picked from commit 9fde1ff5fc114b5edb755ed40944607419b62184) Signed-off-by: Michael Armbrust mich...@databricks.com commit b5b632c8cd02fd1e65ebd22216d20ec76715fc5d Author: Kousuke Saruta saru...@oss.nttdata.co.jp Date: 2014-08-14T00:42:38Z [SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #1891 from sarutak/SPARK-2970 and squashes the following commits: 4a2d2fe [Kousuke Saruta] Modified comment style 8bd833c [Kousuke Saruta] Modified style 6c0997c [Kousuke Saruta] Modified the timing of shutdown hook execution. It should be executed before shutdown hook of o.a.h.f.FileSystem (cherry picked from commit 905dc4b405e679feb145f5e6b35e952db2442e0d) Signed-off-by: Michael Armbrust mich...@databricks.com commit a8d2649719b3d8fdb1eed29ef179a6a896b3e37a Author: guowei guo...@upyoo.com Date: 2014-08-14T00:45:24Z [SPARK-2986] [SQL] fixed: setting properties does not effect it seems that set command does not run by SparkSQLDriver. it runs on hive api. user can not change reduce number by setting spark.sql.shuffle.partitions but i think setting hive properties seems just a role to spark sql. Author: guowei guo...@upyoo.com Closes #1904 from guowei2/temp-branch and squashes the following commits: 7d47dde [guowei] fixed: setting properties like spark.sql.shuffle.partitions does not effective (cherry picked from commit 63d637ca8559d4344d1661500b8ad868bb47) Signed-off-by: Michael Armbrust mich...@databricks.com commit c6cb55a784ba8f9e5c4e7aadcc3ec9dce24f49ee Author: Patrick Wendell pwend
[GitHub] spark pull request: [SPARK-3480] Throws out Not a valid command 'y...
Github user jameszhouyi closed the pull request at: https://github.com/apache/spark/pull/2353 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
GitHub user jameszhouyi opened a pull request: https://github.com/apache/spark/pull/2359 SPARK-3480 - Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' Symptom: Run ./dev/run-tests and dump outputs as following: Running Scala style checks Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ Possible Cause: I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like echo -e q\n | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \ scalastyle.txt echo -e q\n | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \ scalastyle.txt From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'.. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jameszhouyi/spark branch-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2359 commit c7d41ec666cf68f44cf362a3c05ac4d4dc9e74ab Author: jameszhouyi james_zhouyi2...@126.com Date: 2014-09-11T07:40:17Z SPARK-3480 - Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
Github user jameszhouyi closed the pull request at: https://github.com/apache/spark/pull/2359 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3480 - Throws out Not a valid command 'y...
GitHub user jameszhouyi reopened a pull request: https://github.com/apache/spark/pull/2359 SPARK-3480 - Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' Symptom: Run ./dev/run-tests and dump outputs as following: Running Scala style checks Scalastyle checks failed at following occurrences: [error] Expected ID character [error] Not a valid command: yarn-alpha [error] Expected project ID [error] Expected configuration [error] Expected ':' (if selecting a configuration) [error] Expected key [error] Not a valid key: yarn-alpha [error] yarn-alpha/scalastyle [error] ^ Possible Cause: I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like echo -e q\n | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \ scalastyle.txt echo -e q\n | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \ scalastyle.txt From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'.. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jameszhouyi/spark branch-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2359 commit c7d41ec666cf68f44cf362a3c05ac4d4dc9e74ab Author: jameszhouyi james_zhouyi2...@126.com Date: 2014-09-11T07:40:17Z SPARK-3480 - Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
