dhealy05 opened a new issue #6146: Pulsar SQL Cluster Fails Out of the Box with Default Deployment v2.5.0 URL: https://github.com/apache/pulsar/issues/6146 With a default deployment of 3 Zookeeper nodes, 3 Broker/BookKeeper nodes, and 3 Pulsar SQL nodes, SQL query "select * from pulsar."public/default".generator_test;" fails. "select * from system.runtime.nodes;" works correctly and "show tables in pulsar."public/default";" works correctly. There are two ways in which it fails: 1. For a cluster with multiple hosts, where conf/presto/catalog/pulsar.properties "pulsar.broker-service-url=http://10.0.0.7:8080,10.0.0.8:8080,10.0.0.9:8080";, the following error message is shown at the Presto CLI with --debug: Failed to get topics in schema public/default: For input string: "8080,10.0.0.8:8080,10.0.0.9:8080" at org.apache.pulsar.sql.presto.PulsarMetadata.getTableMetadata(PulsarMetadata.java:298) at org.apache.pulsar.sql.presto.PulsarMetadata.getTableMetadata(PulsarMetadata.java:159) at com.facebook.presto.metadata.MetadataManager.getTableMetadata(MetadataManager.java:377) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:841) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:257) at com.facebook.presto.sql.tree.Table.accept(Table.java:53) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:269) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:1787) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:944) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:257) at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:127) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:269) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:279) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:675) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:257) at com.facebook.presto.sql.tree.Query.accept(Query.java:94) at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27) at com.facebook.presto.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:269) at com.facebook.presto.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:243) at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:72) at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:64) at com.facebook.presto.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:169) at com.facebook.presto.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:677) at com.facebook.presto.execution.SqlQueryManager.createQueryInternal(SqlQueryManager.java:444) at com.facebook.presto.execution.SqlQueryManager.lambda$createQuery$3(SqlQueryManager.java:377) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.pulsar.client.admin.PulsarAdminException: java.lang.NumberFormatException: For input string: "8080,X.X.X.2:8080,X.X.X.3:8080". 2. If we instead switch to "pulsar.broker-service-url=http://X.X.X.1:8080"; by itself, the following is shown on the Coordinator Node: 2020-01-26T20:04:50.168Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-01-26T20:04:50.169Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:java.io.tmpdir=/tmp 2020-01-26T20:04:50.169Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:java.compiler=<NA> 2020-01-26T20:04:50.169Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.name=Linux 2020-01-26T20:04:50.169Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.arch=amd64 2020-01-26T20:04:50.169Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.version=4.14.154-128.181.amzn2.x86_64 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:user.name=ec2-user 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:user.home=/home/ec2-user 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:user.dir=/home/ec2-user/apache-pulsar-2.5.0/lib/presto 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.memory.free=88MB 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.memory.max=16384MB 2020-01-26T20:04:50.170Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Client environment:os.memory.total=256MB 2020-01-26T20:04:50.213Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=10.0.0.4:2181,10.0.0.5:2181,10.0.0.6:2181 sessionTimeout=10000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@6f62d802 2020-01-26T20:04:50.224Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2020-01-26T20:04:50.231Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2020-01-26T20:04:50.242Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.zookeeper.ClientCnxn zookeeper.request.timeout value is 0. feature enabled= 2020-01-26T20:04:50.273Z INFO Query-20200126_200449_00001_av2iy-144-SendThread(10.0.0.6:2181) org.apache.zookeeper.ClientCnxn Opening socket connection to server ip-10-0-0-6.us-west-2.compute.internal/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error) 2020-01-26T20:04:50.274Z INFO Query-20200126_200449_00001_av2iy-144-SendThread(10.0.0.6:2181) org.apache.zookeeper.ClientCnxn Socket connection established, initiating session, client: /10.0.0.10:53282, server: ip-10-0-0-6.us-west-2.compute.internal/10.0.0.6:2181 2020-01-26T20:04:50.284Z INFO Query-20200126_200449_00001_av2iy-144-SendThread(10.0.0.6:2181) org.apache.zookeeper.ClientCnxn Session establishment complete on server ip-10-0-0-6.us-west-2.compute.internal/10.0.0.6:2181, sessionid = 0x300000aa5d30009, negotiated timeout = 10000 2020-01-26T20:04:50.288Z INFO Query-20200126_200449_00001_av2iy-144-EventThread org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase ZooKeeper client is connected now. 2020-01-26T20:04:50.338Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-01-26T20:04:50.339Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-01-26T20:04:50.339Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.meta.MetadataDrivers BookKeeper metadata driver manager initialized 2020-01-26T20:04:50.343Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase Initialize zookeeper metadata driver with external zookeeper client : ledgersRootPath = /ledgers. 2020-01-26T20:04:50.367Z WARN Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.util.EventLoopUtil Could not use Netty Epoll event loop: failed to load the required native library 2020-01-26T20:04:50.400Z ERROR Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver. 2020-01-26T20:04:50.408Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Initialize rackaware ensemble placement policy @ <Bookie:10.0.0.10:0> @ /default-rack : org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver. 2020-01-26T20:04:50.408Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl Not weighted 2020-01-26T20:04:50.413Z INFO Query-20200126_200449_00001_av2iy-144 org.apache.bookkeeper.client.BookKeeper Weighted ledger placement is not enabled 2020-01-26T20:04:50.446Z INFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/10.0.0.8:3181 2020-01-26T20:04:50.469Z INFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/10.0.0.9:3181 2020-01-26T20:04:50.483Z INFO BookKeeperClientScheduler-OrderedScheduler-0-0 org.apache.bookkeeper.net.NetworkTopologyImpl Adding a new node: /default-rack/10.0.0.7:3181 2020-01-26T20:04:50.486Z ERROR Query-20200126_200449_00001_av2iy-144 org.apache.pulsar.sql.presto.PulsarSplitManager Failed to get splits java.io.IOException: Failed to initialize ledger manager factory at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:520) at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:368) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$DefaultBkFactory.<init>(ManagedLedgerFactoryImpl.java:183) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:122) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:114) at org.apache.pulsar.sql.presto.PulsarConnectorCache.initManagedLedgerFactory(PulsarConnectorCache.java:108) at org.apache.pulsar.sql.presto.PulsarConnectorCache.<init>(PulsarConnectorCache.java:66) at org.apache.pulsar.sql.presto.PulsarConnectorCache.getConnectorCache(PulsarConnectorCache.java:83) at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplitsNonPartitionedTopic(PulsarSplitManager.java:224) at org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:126) at com.facebook.presto.split.SplitManager.getSplits(SplitManager.java:64) at com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:146) at com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:122) at com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:136) at com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:108) at com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:113) at com.facebook.presto.sql.planner.DistributedExecutionPlanner.plan(DistributedExecutionPlanner.java:85) at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:385) at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:287) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to initialized ledger manager factory at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:243) at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:518) ... 21 more Caused by: java.io.IOException: Empty Ledger Root Path. at org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.newLedgerManagerFactory(AbstractZkLedgerManagerFactory.java:158) at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:239) ... 22 more 2020-01-26T20:04:50.510Z INFO query-execution-1 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20200126_200449_00001_av2iy :: Transaction:[b7ea1383-3bb7-49d6-87bb-65a1f40e531b] :: elapsed 959ms :: planning 499ms :: scheduling 460ms :: running 0ms :: finishing 460ms :: begin 2020-01-26T20:04:49.529Z :: end 2020-01-26T20:04:50.488Z This cluster has not produced or consumed any messages/topics besides those generated by CLI test/data generator. All offloaders and connectors are downloaded on both the SQL nodes and the broker/BookKeeper nodes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
