Shenfeng1011 opened a new issue #7639:
URL: https://github.com/apache/pulsar/issues/7639
can not offload data to hdfs?
version:2.6.0
1.broker.conf
managedLedgerOffloadDriver=filesystem
fileSystemProfilePath="../conf/filesystem_offload_core_site.xml"
fileSystemURI="hdfs://10.130.161.123:8020"
2.filesystem_offload_core_site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://10.130.161.123:8020</value>
</property>
3.offloaders directory content
tiered-storage-file-system-2.6.0.nar
tiered-storage-jcloud-2.6.0.nar
operate step
4.pulsar standalone
5.pulsar-admin namespaces set-offload-threshold --size 2K public/default
the log is:
1:28:01.723 [pulsar-web-69-4] INFO
org.apache.pulsar.broker.admin.impl.NamespacesBase - [null] Successfully
updated offloadThreshold configuration: namespace=public/default, value=2048
11:28:01.723 [pulsar-ordered-OrderedExecutor-1-0-EventThread] INFO
org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000
sessionid:0x1000d0ae910000a local:/127.0.0.1:44442
remoteserver:localhost/127.0.0.1:2181 lastZxid:5104 xid:80 sent:174 recv:176
queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/admin/policies/public/default
11:28:01.724 [pulsar-web-69-4] INFO org.eclipse.jetty.server.RequestLog -
127.0.0.1 - - [23/7月/2020:11:28:01 +0800] "PUT
/admin/v2/namespaces/public/default/offloadThreshold HTTP/1.1" 204 0 "-"
"Pulsar-Java-v2.5.2" 40
11:28:01.725 [ForkJoinPool.commonPool-worker-1] INFO
org.apache.pulsar.broker.service.BrokerService - /admin/policies/public/default
updating with
Policies{auth_policies=org.apache.pulsar.common.policies.data.AuthPolicies@745f,
replication_clusters=[standalone], bundles=BundlesData{numBundles=4,
boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xffffffff]},
backlog_quota_map={}, persistence=null, deduplicationEnabled=null,
autoTopicCreationOverride=null, autoSubscriptionCreationOverride=null,
clusterDispatchRate={}, topicDispatchRate={}, subscriptionDispatchRate={},
replicatorDispatchRate={}, clusterSubscribeRate={}, publishMaxMessageRate={},
latency_stats_sample_rate={}, antiAffinityGroup=null, message_ttl_in_seconds=0,
subscription_expiration_time_minutes=0, retention_policies=null, deleted=false,
encryption_required=false, delayed_delivery_policies=null,
subscription_auth_mode=None, max_producers_per_topic=0,
max_consumers_per_topic=0, max_consumers_per_subscription
=0, max_unacked_messages_per_consumer=-1,
max_unacked_messages_per_subscription=-1, compaction_threshold=0,
offload_threshold=2048, offload_deletion_lag_ms=null,
schema_auto_update_compatibility_strategy=Full,
schema_validation_enforced=false, schema_compatibility_Strategy=UNDEFINED,
is_allow_auto_update_Schema=true,
offload_policies=OffloadPolicies{managedLedgerOffloadDriver=null,
managedLedgerOffloadMaxThreads=2, managedLedgerOffloadPrefetchRounds=1,
managedLedgerOffloadThresholdInBytes=2048,
managedLedgerOffloadDeletionLagInMillis=null,
s3ManagedLedgerOffloadRegion=null, s3ManagedLedgerOffloadBucket=null,
s3ManagedLedgerOffloadServiceEndpoint=null,
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864,
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576,
s3ManagedLedgerOffloadRole=null,
s3ManagedLedgerOffloadRoleSessionName=pulsar-s3-offload,
gcsManagedLedgerOffloadRegion=null, gcsManagedLedgerOffloadBucket=null,
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864, gcsManagedLedgerO
ffloadReadBufferSizeInBytes=1048576,
gcsManagedLedgerOffloadServiceAccountKeyFile=null, fileSystemProfilePath=null,
fileSystemURI=null}}
6.publish some message more than 2k but in hdfs found nothing
7.bin/pulsar-admin topics offload --size-threshold 1K
public/default/my-vehicle-string-topic3
11:45:41.081 [pulsar-web-69-11] INFO
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[persistent://public/default/my-vehicle-string-topic3] Starting offload
operation at messageId 481:0:-1
11:45:41.081 [pulsar-web-69-11] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/default/persistent/my-vehicle-string-topic3] Start ledgersOffload.
ledgers=[481] totalSize=1800
11:45:41.082 [pulsar-web-69-11] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[public/default/persistent/my-vehicle-string-topic3] No ledgers to offload
11:45:41.082 [pulsar-web-69-11] INFO
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[persistent://public/default/my-vehicle-string-topic3] Completed successfully
offload operation at messageId 481:0:-1
still found nothing in hdfs
8.bin/pulsar-admin namespaces policies public/default
found
"offload_policies" : {
"offloadersDirectory" : "./offloaders",
"managedLedgerOffloadMaxThreads" : 2,
"managedLedgerOffloadPrefetchRounds" : 1,
"managedLedgerOffloadThresholdInBytes" : 2048,
"s3ManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"s3ManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"s3ManagedLedgerOffloadRoleSessionName" : "pulsar-s3-offload",
"gcsManagedLedgerOffloadMaxBlockSizeInBytes" : 67108864,
"gcsManagedLedgerOffloadReadBufferSizeInBytes" : 1048576,
"gcsDriver" : false,
"fileSystemDriver" : false,
"s3Driver" : false
}
8.it seams that fileSystemDriver is not loaded. where was i wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
