lichaohao opened a new pull request, #3610:
URL: https://github.com/apache/amoro/pull/3610
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in
https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title,
e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related
issue
-->
Close #3586
## Brief change log
#my external sparkOptimizer command
export HADOOP_USER_NAME=hive && export HADOOP_CONF_DIR=/etc/hadoop/conf/ &&
export SPARK_CONF_DIR=/etc/spark3/conf/ &&
/data/hadoop22/spark-3.3.4/bin/spark-submit
--class org.apache.amoro.optimizer.spark.SparkOptimizer
--master yarn --deploy-mode=cluster --driver-memory 10g --executor-memory
30g --executor-cores 2
--num-executors 20 --name amoro-spark-optimizer
--conf spark.dynamicAllocation.shuffleTracking.enabled=true
--conf spark.executor.userClassPathFirst=true
--conf spark.dynamicAllocation.maxExecutors=20
--conf spark.driver.userClassPathFirst=true
--conf spark.dynamicAllocation.enabled=true
--conf spark.yarn.queue=flink-amoro --proxy-user hadoop
/data/hadoop22/master_amoro/amoro-0.8-SNAPSHOT/plugin/optimizer/spark/optimizer-job.jar
-a zookeeper://zyxfcdp01:2181,zyxfcdp02:2181,zyxfcdp03:2181/tancha-amoro -p
20 -g tancha-amoro
amoro-config.yaml
#High availability configuration.
ha:
enabled: true
cluster-name: tancha-amoro
zookeeper-address: zyxfcdp01:2181,zyxfcdp02:2181,zyxfcdp03:2181
version: 0.8.0
25/05/30 11:34:33 WARN client.ThriftClientPool: Maybe server is restarting,
wait a while
25/05/30 11:34:39 ERROR common.AbstractOptimizerOperator: Call ams got an
error and will try again later
org.apache.amoro.client.ThriftException: Client can not connect.
at
org.apache.amoro.client.ThriftClientPool.iface(ThriftClientPool.java:209)
at
org.apache.amoro.client.OptimizingClientPools.getClient(OptimizingClientPools.java:38)
at
org.apache.amoro.optimizer.common.AbstractOptimizerOperator.callAms(AbstractOptimizerOperator.java:53)
at
org.apache.amoro.optimizer.common.OptimizerToucher.checkToken(OptimizerToucher.java:74)
at
org.apache.amoro.optimizer.common.OptimizerToucher.start(OptimizerToucher.java:59)
at
org.apache.amoro.optimizer.common.Optimizer.startOptimizing(Optimizer.java:65)
at
org.apache.amoro.optimizer.spark.SparkOptimizer.main(SparkOptimizer.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
25/05/30 11:34:44 WARN client.ThriftClientPool: Maybe server is restarting,
wait a while
25/05/30 11:34:46 WARN client.ThriftClientPool: Maybe server is restarting,
wait a while
25/05/30 11:34:46 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint:
Registered executor NettyRpcEndpointRef(spark-
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] Run test locally before making a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
