Re: acquire and give back resources dynamically
http://spark.apache.org/docs/latest/running-on-yarn.html Spark just a Yarn application > 在 2014年8月14日，11:12，牛兆捷 写道： > > Dear all: > > Does spark can acquire resources from and give back resources to > YARN dynamically ? > > > -- > *Regards,* > *Zhaojie* - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: fair scheduler
@Crystal
You can use spark on yarn. Yarn have fair scheduler,modified yarn-site.xml.
发自我的 iPad
> 在 2014年8月11日，6:49，Matei Zaharia 写道：
>
> Hi Crystal,
>
> The fair scheduler is only for jobs running concurrently within the same
> SparkContext (i.e. within an application), not for separate applications on
> the standalone cluster manager. It has no effect there. To run more of those
> concurrently, you need to set a cap on how many cores they each grab with
> spark.cores.max.
>
> Matei
>
> On August 10, 2014 at 12:13:08 PM, 李宜芳 (xuite...@gmail.com) wrote:
>
> Hi
>
> I am trying to switch from FIFO to FAIR with standalone mode.
>
> my environment:
> hadoop 1.2.1
> spark 0.8.0 using stanalone mode
>
> and i modified the code..
>
> ClusterScheduler.scala -> System.getProperty("spark.scheduler.mode",
> "FAIR"))
> SchedulerBuilder.scala ->
> val DEFAULT_SCHEDULING_MODE = SchedulingMode.FAIR
>
> LocalScheduler.scala ->
> System.getProperty("spark.scheduler.mode", "FAIR)
>
> spark-env.sh ->
> export SPARK_JAVA_OPTS="-Dspark.scheduler.mode=FAIR"
> export SPARK_JAVA_OPTS=" -Dspark.scheduler.mode=FAIR" ./run-example
> org.apache.spark.examples.SparkPi spark://streaming1:7077
>
>
> but it's not work
> i want to switch from fifo to fair
> how can i do??
>
> Regards
> Crystal Lee
>
-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
[GitHub] spark pull request: replace println to log4j
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1372#issuecomment-50223879 My account is fireflyc, please assign the issue to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fixed the number of worker thread
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1485#issuecomment-49501533 My program is spark streaming over Hadoop yarn.It work for user click stream. I read code,number of worker threads and block? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fixed the number of worker thread
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1485#issuecomment-49495043 My application have 1000+ Worker Threads.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fixed the number of worker thread
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1485#issuecomment-49493529 Will always have a task to run, when the system is idle all threads will not participate CPU. When need run task need not new a thread. `fixed` is great way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fixed the number of worker thread
GitHub user fireflyc opened a pull request: https://github.com/apache/spark/pull/1485 Fixed the number of worker thread There are a lot of input Block cause too many Worker threads and will load all data.So it should be to control the number of Worker threads You can merge this pull request into a Git repository by running: $ git pull https://github.com/fireflyc/spark fixed-executor-thread Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1485.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1485 commit 1facd581b3e1e37cc896a7db8d3bb8e9ab088686 Author: fireflyc Date: 2014-07-18T15:19:46Z Fixed the number of worker thread There are a lot of input Block cause too many Worker threads and will load all data.So it should be to control the number of Worker threads --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: replace println to log4j
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1372#issuecomment-49125597 I modified 'info' into the 'debug' level. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: replace println to log4j
Github user fireflyc commented on the pull request: https://github.com/apache/spark/pull/1372#issuecomment-48739929 I have verified, the log level is set to Info right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: replace println to log4j
GitHub user fireflyc opened a pull request: https://github.com/apache/spark/pull/1372 replace println to log4j Our program needs to receive a large amount of data and run for a long time. We set the log level to WARN but "Storing iterator" "received single" as such message written to the log file. (over yarn) You can merge this pull request into a Git repository by running: $ git pull https://github.com/fireflyc/spark fix-replace-stdout-log Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1372 commit fa22a3879a9d4887cd229966b6523dd65b2f6003 Author: fireflyc Date: 2014-07-11T08:03:20Z replace println to log4j Our program needs to receive a large amount of data and run for a long time. We set the log level to WARN but "Storing iterator" "received single" as such message written to the log file. (over yarn) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jboss-user] [JBoss Portal] - Re: JBoss Portlet Container 2.0 released
jboss portlet container can run on weblogic or webshpereï¼ View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4196453#4196453 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4196453 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
