HSLife1991 commented on issue #6927:
URL: https://github.com/apache/seatunnel/issues/6927#issuecomment-2141341989
it works when add hdfs_site_path.
another issue, does map have hdfs ha configuration in hazelcast.yaml?
my config is like this and it does not work:
hazelcast:
cluster-name: seatunnel
network:
rest-api:
enabled: true
endpoint-groups:
CLUSTER_WRITE:
enabled: true
DATA:
enabled: true
join:
tcp-ip:
enabled: true
member-list:
- bc-gcp-sg-node03.c.proj.internal
- bc-gcp-sg-node04.c.proj.internal
- bc-gcp-sg-node05.c.proj.internal
port:
auto-increment: false
port: 5801
properties:
hazelcast.invocation.max.retry.count: 20
hazelcast.tcp.join.port.try.count: 30
hazelcast.logging.type: log4j2
hazelcast.operation.generic.thread.count: 50
map:
engine*:
map-store:
enabled: true
initial-mode: EAGER
factory-class-name:
org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
properties:
type: hdfs
namespace: /tmp/seatunnel/imap
clusterName: seatunnel
storage.type: hdfs
#fs.defaultFS: hdfs://bc-gcp-sg-node02.c.proj.internal:8020 # it
works when use this active namenode
fs.defaultFS: hdfs://prod # Ensure that the directory has written
permission
seatunnel.hadoop.dfs.nameservices: prod
seatunnel.hadoop.dfs.ha.namenodes.prod: nn1,nn2
seatunnel.hadoop.dfs.namenode.rpc-address.prod.nn1:
bc-gcp-sg-node01.c.proj.internal:8020
seatunnel.hadoop.dfs.namenode.rpc-address.prod.nn2:
bc-gcp-sg-node02.c.proj.internal:8020
seatunnel.hadoop.dfs.client.failover.proxy.provider.prod:
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
report error when config with hdfs ha:
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at
org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.util.concurrent.CompletionException:
org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: Can not
get coordinator service from an active master node.
at
com.hazelcast.spi.impl.AbstractInvocationFuture.wrapInCompletionException(AbstractInvocationFuture.java:1347)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.cascadeException(AbstractInvocationFuture.java:1340)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.access$200(AbstractInvocationFuture.java:65)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
