richard-scott opened a new issue, #218:
URL: https://github.com/apache/bifromq/issues/218
## Summary
BifroMQ doesn't seem to enforce tenant-level settings. My plugin is being
called, returning correct values, and successfully casting/returning them.
BifroMQ accepts the values and doesn't enforce them.
**Critical Finding**: All tested tenant-level settings (6 out of 6) are
**NOT being enforced**, suggesting a systemic issue in which tenant-level
settings from the Setting Provider plugin are not enforced by BifroMQ, even
though the plugin is working correctly.
## Successfully Created Plugins
For context, I have successfully created and migrated all 5 BifroMQ plugins
to the Maven archetype structure:
1. ✅ **Auth Provider**
(`com.bifromq.plugin.authprovider.CustomAuthProvider`) - Fully functional
- Handles authentication and authorisation
- Supports both MQTT 3.1.1 and MQTT 5.0
- Enforces ACL rules correctly
- **Status**: Working correctly
2. ✅ **Resource Throttler**
(`com.bifromq.plugin.resourcethrottler.CustomResourceThrottler`) - Fully
functional
- Enforces connection limits (max_connections)
- Enforces rate limits (messages per second)
- Enforces bandwidth limits (bytes per second)
- **Status**: Working correctly
3. ⚠️ **Setting Provider**
(`com.bifromq.plugin.settingprovider.CustomSettingProvider`) - **Plugin works,
but BifroMQ doesn't enforce limits**
- Plugin correctly implements `ISettingProvider` interface
- Plugin returns correct values for all settings
- The plugin is called by BifroMQ, and values are returned successfully
- **Status**: Plugin implementation is correct, but BifroMQ does not
enforce the provided settings (this issue)
4. ✅ **Client Balancer**
(`com.bifromq.plugin.clientbalancer.CustomClientBalancer`) - Fully functional
- Implements custom client balancing strategies
- **Status**: Working correctly
5. ✅ **Event Collector**
(`com.bifromq.plugin.eventcollector.CustomEventCollector`) - Fully functional
- Collects runtime events from BifroMQ
- Tracks active connections per tenant
- **Status**: Working correctly
**Note**: All plugins follow the Maven archetype structure with `conf/`
directories, `PluginContext` classes, and proper `pom.xml` configuration. The
Setting Provider plugin is the only one where we've found that BifroMQ isn't
enforcing the tenant-level settings it provides.
## Affected Settings
The following tenant-level settings are **NOT being enforced** (tested and
confirmed):
1. **MaximumQoS** - Maximum QoS level allowed
2. **MaxTopicLength** - Maximum topic name length
3. **MaxUserPayloadBytes** - Maximum user payload size
4. **RetainEnabled** - Whether retain messages are allowed
5. **WildcardSubscriptionEnabled** - Whether wildcard subscriptions are
allowed
6. **MaxTopicLevelLength** - Maximum length of each topic level
### Settings Not Yet Tested
The following settings from the [official
documentation](https://bifromq.apache.org/docs/plugin/setting_provider/tenantsetting/)
have not been tested but may also not be enforced:
- **SharedSubscriptionEnabled** - Whether shared subscriptions are allowed
- **MQTT3Enabled/MQTT4Enabled/MQTT5Enabled** - Protocol version enablement
- **MaxSessionExpirySeconds** - Maximum session expiry time
- **SessionInboxSize** - Maximum size of session inbox
- **MaxTopicLevels** - Maximum number of topic levels
- **MaxTopicAlias** - Maximum number of topic aliases
- **MaxSharedGroupMembers** - Maximum members in a shared subscription group
- **MaxTopicFiltersPerInbox** - Maximum topic filters per inbox
- **MsgPubPerSec** - Maximum messages published per second per connection
- **ReceivingMaximum** - Maximum receiving messages per connection
- **InBoundBandWidth/OutBoundBandWidth** - Maximum bandwidth limits
- And others...
**Note**: Based on the pattern observed, it appears that **most or all
tenant-level settings may not be enforced**, even though the plugin is
correctly providing them.
## Expected Behaviour
According to [BifroMQ's official
documentation](https://bifromq.apache.org/docs/plugin/setting_provider/tenantsetting/):
- **MaximumQoS**: "Sets the maximum QoS level. Valid values: 0, 1, 2." -
Should reject PUBLISH with QoS exceeding limit
- **MaxTopicLength**: "Maximum total length of a topic." - Should reject
topics exceeding this length
- **MaxUserPayloadBytes**: "Maximum user payload size in bytes." - Should
reject packets exceeding this size
- **RetainEnabled**: "Enables or disables message retain feature." - Should
reject PUBLISH with retain flag if disabled
## Actual Behaviour
The Setting Provider plugin is correctly providing these values, but BifroMQ
is not enforcing them:
### Test Results
**Tenant Configuration** (`conf/tenants.json`):
```json
{
"tenants": {
"tenant2": {
"settings": {
"MaxTopicLength": 128,
"MaxPacketSizeBytes": 32768,
"MaxQoS": 1,
"RetainEnabled": false,
"WildcardSubscriptionEnabled": false,
"MaxTopicLevelLength": 10
}
}
}
}
```
**Test Results**:
- ❌ **MaxTopicLength**: 150-byte topic accepted (limit: 128 bytes) - Should
be rejected
- ❌ **MaxPacketSizeBytes**: 35000-byte packet accepted (limit: 32768 bytes)
- Should be rejected
- ❌ **MaxQoS**: QoS 2 publish accepted (limit: MaxQoS=1) - Should be rejected
- ❌ **RetainEnabled**: Retain flag accepted (setting: RetainEnabled=false) -
Should be rejected
- ❌ **WildcardSubscriptionEnabled**: Wildcard subscription accepted
(setting: WildcardSubscriptionEnabled=false) - Should be rejected
- ❌ **MaxTopicLevelLength**: Topic with 17-byte level accepted (limit: 10
bytes per level) - Should be rejected
All test cases show `Publish RC: 0 (Success)`, even though they should be
rejected.
## Evidence: Plugin is Working Correctly
Enhanced logging shows the plugin is functioning correctly:
### 1. BifroMQ Calls the Plugin
```
[CustomSettingProvider] provide() CALLED by BifroMQ: setting=MaximumQoS,
tenant=tenant2
[CustomSettingProvider] provide() CALLED by BifroMQ: setting=MaxTopicLength,
tenant=tenant2
[CustomSettingProvider] provide() CALLED by BifroMQ:
setting=MaxUserPayloadBytes, tenant=tenant2
```
### 2. Plugin Returns Correct Values
```
[CustomSettingProvider] provide() RETURNING VALUE: setting=MaximumQoS,
tenant=tenant2, value=1 (type: Integer) - BifroMQ SHOULD USE THIS VALUE
[CustomSettingProvider] provide() RETURNING VALUE: setting=MaxTopicLength,
tenant=tenant2, value=128 (type: Integer) - BifroMQ SHOULD USE THIS VALUE
[CustomSettingProvider] provide() RETURNING VALUE:
setting=MaxUserPayloadBytes, tenant=tenant2, value=32768 (type: Integer) -
BifroMQ SHOULD USE THIS VALUE
```
### 3. Plugin Successfully Returns Values
```
[CustomSettingProvider] provide() SUCCESSFULLY CAST AND RETURNED:
setting=MaximumQoS, tenant=tenant2, result=1
[CustomSettingProvider] provide() SUCCESSFULLY CAST AND RETURNED:
setting=MaxTopicLength, tenant=tenant2, result=128
[CustomSettingProvider] provide() SUCCESSFULLY CAST AND RETURNED:
setting=MaxUserPayloadBytes, tenant=tenant2, result=32768
```
### 4. But BifroMQ Ignores Them
Test output shows violations are accepted:
```
[✗] MaxQoS: QoS 2 was accepted (should be rejected)
Publish RC: 0 (0 = Success, but should be rejected)
Requested QoS: 2 (MaxQoS limit: 1)
[✗] MaxTopicLevelLength: Topic with long level was accepted (should be
rejected)
Publish RC: 0 (0 = Success, but should be rejected)
Topic: test/verylonglevelname/abc
Longest Level: 'verylonglevelname' (17 bytes, limit: 10 bytes)
```
**Pattern Observed**: All 6 tested settings show identical behaviour -
plugin provides correct values, but BifroMQ does not enforce them.
## Configuration
### BifroMQ Version
- Version: `4.0.0-incubating-RC3`
### Setting Provider Plugin
- Plugin FQN: `com.bifromq.plugin.settingprovider.CustomSettingProvider`
- Status: ✅ Enabled and loaded
- Implementation: Correctly implements `ISettingProvider` interface
### Cache Configuration
Configured in `docker-compose.yml`:
```yaml
JVM_HEAP_OPTS: ... -Dsetting_provide_init_value=true
-Dsetting_refresh_seconds=1 -Dsetting_expire_seconds=1
-Dsetting_tenant_cache_limit=100
```
### Plugin Reload Mechanism
- Reload interval: 60 seconds or on file modification
- File path: `/home/bifromq/conf/tenants.json`
- Status: ✅ Working (settings reloaded successfully)
## Steps to Reproduce
1. **Configure tenant settings** in `conf/tenants.json`:
```json
{
"tenants": {
"tenant2": {
"settings": {
"MaxTopicLength": 128,
"MaxPacketSizeBytes": 32768,
"MaxQoS": 1,
"RetainEnabled": false,
"WildcardSubscriptionEnabled": false,
"MaxTopicLevelLength": 10
}
}
}
}
```
2. **Start BifroMQ cluster** with Setting Provider plugin enabled
3. **Run test suite**:
```bash
python3 bin/test_tenant_settings.py \
--host localhost \
--port 8883 \
--tenant tenant2 \
--user user1 \
--password tenant2_user1_pass123 \
--protocol 5
```
4. **Observe results**: All limit violations are accepted instead of rejected
## Logs
### Plugin Logs (showing correct behaviour)
```
[CustomSettingProvider] provide() CALLED by BifroMQ: setting=MaximumQoS,
tenant=tenant2
[CustomSettingProvider] provide() RETURNING VALUE: setting=MaximumQoS,
tenant=tenant2, value=1 (type: Integer) - BifroMQ SHOULD USE THIS VALUE
[CustomSettingProvider] provide() SUCCESSFULLY CAST AND RETURNED:
setting=MaximumQoS, tenant=tenant2, result=1
```
### Test Output (showing enforcement failure)
```
[✗] MaxQoS: QoS 2 was accepted (should be rejected)
Publish RC: 0 (0 = Success, but should be rejected)
Requested QoS: 2 (MaxQoS limit: 1)
```
## Impact
This issue prevents proper tenant-level resource control and security
policies:
- **Security**: Cannot enforce QoS restrictions per tenant
- **Resource Control**: Cannot limit topic length, topic level length, or
payload size per tenant
- **Feature Control**: Cannot disable retain messages, wildcard
subscriptions, or other features per tenant
- **Multi-tenancy**: Breaks tenant isolation expectations - all tenants
effectively use default settings
- **Compliance**: Cannot enforce tenant-specific policies required for
regulatory compliance
- **Resource Management**: Cannot prevent resource exhaustion by limiting
tenant capabilities
**Severity**: **HIGH** - This affects the core functionality of tenant-level
settings, which is a fundamental feature for multi-tenant deployments.
## References
- [BifroMQ Setting Provider
Documentation](https://bifromq.apache.org/docs/plugin/setting_provider/)
- [Tenant-level Settings
Reference](https://bifromq.apache.org/docs/plugin/setting_provider/tenantsetting/)
## Additional Information
### Plugin Implementation Details
The plugin correctly:
- ✅ Implements `ISettingProvider` interface
- ✅ Returns correct values for all settings
- ✅ Handles type casting correctly
- ✅ Reloads settings every 60 seconds or on file modification
- ✅ Logs all calls and return values (enhanced logging added for debugging)
### Complete Plugin Code
The complete `provide()` method and related helper methods from
`CustomSettingProvider.java`:
```java
/**
* Provide a setting value for a specific tenant.
*
* This method is called by BifroMQ's worker threads to get the current value
* of a setting for a tenant. The method must be lightweight and
non-blocking.
*
* @param <R> The return type (depends on the Setting type - Integer,
Boolean, etc.)
* @param setting The setting to provide a value for (from Setting enum)
* @param tenantId The tenant identifier
* @return The setting value (must match the Setting's expected type), or
null to use the current/default value
*/
@Override
public <R> R provide(Setting setting, String tenantId) {
// Get setting name from the Setting enum
String settingName = setting.name();
// Log every call to provide() - this proves BifroMQ is calling the
plugin
System.err.println(String.format("[CustomSettingProvider] provide()
CALLED by BifroMQ: setting=%s, tenant=%s",
settingName, tenantId));
// Get value from configuration (tenant-specific or default)
Object value = getSettingValue(settingName, tenantId);
// Log the value being returned - this proves we're returning the
correct value
if (value != null) {
System.err.println(String.format("[CustomSettingProvider] provide()
RETURNING VALUE: setting=%s, tenant=%s, value=%s (type: %s) - BifroMQ SHOULD
USE THIS VALUE",
settingName, tenantId, value, value.getClass().getSimpleName()));
try {
R result = (R) value;
System.err.println(String.format("[CustomSettingProvider]
provide() SUCCESSFULLY CAST AND RETURNED: setting=%s, tenant=%s, result=%s",
settingName, tenantId, result));
return result;
} catch (ClassCastException e) {
// Type mismatch - return null to use default
System.err.println(String.format("[CustomSettingProvider]
provide() TYPE MISMATCH: setting=%s, tenant=%s, error=%s - returning null",
settingName, tenantId, e.getMessage()));
return null;
}
} else {
// Log when no value is found (only for tenant2 to reduce noise)
if ("tenant2".equals(tenantId)) {
System.err.println(String.format("[CustomSettingProvider]
provide() RETURNING NULL: setting=%s, tenant=%s - BifroMQ will use default",
settingName, tenantId));
}
}
// Return null to use BifroMQ's default/current value
return null;
}
/**
* Get setting value for a tenant, checking tenant-specific settings first,
* then defaults, then returning null to use BifroMQ's default.
*/
private Object getSettingValue(String settingName, String tenantId) {
loadSettingsIfNeeded();
// Get the JSON field name (may differ from Setting enum name)
String jsonFieldName = getJsonFieldName(settingName);
// Check tenant-specific settings first (try "Bytes" suffix version)
Map<String, Object> tenantSettings = tenantSettingsCache.get(tenantId);
if (tenantSettings != null && tenantSettings.containsKey(jsonFieldName))
{
return tenantSettings.get(jsonFieldName);
}
// Also check for original field name (backward compatibility)
if (tenantSettings != null && tenantSettings.containsKey(settingName)) {
return tenantSettings.get(settingName);
}
// Check default settings (try "Bytes" suffix version)
if (defaultSettings.containsKey(jsonFieldName)) {
return defaultSettings.get(jsonFieldName);
}
// Also check for original field name in defaults (backward
compatibility)
if (defaultSettings.containsKey(settingName)) {
return defaultSettings.get(settingName);
}
// Return null to use BifroMQ's default
return null;
}
/**
* Load settings from configuration file.
* This method is called lazily and caches the results.
*/
private void loadSettingsIfNeeded() {
long currentTime = System.currentTimeMillis();
File settingsFile = new File(SETTINGS_FILE_PATH);
// Check if file exists and if we need to reload
if (!settingsFile.exists()) {
return; // No settings file, use defaults
}
long fileModified = settingsFile.lastModified();
boolean needsReload = (currentTime - lastReloadTime >
RELOAD_INTERVAL_MS) ||
(fileModified > lastModified);
if (!needsReload) {
return; // Cache is still valid
}
try (InputStream is = new FileInputStream(settingsFile)) {
JsonNode root = objectMapper.readTree(is);
// Load default settings
if (root.has("defaults") && root.get("defaults").has("settings")) {
JsonNode defaults = root.get("defaults").get("settings");
defaultSettings.clear();
defaults.fields().forEachRemaining(entry -> {
String key = entry.getKey();
JsonNode value = entry.getValue();
if (!value.isNull()) {
defaultSettings.put(key, parseValue(value));
}
});
}
// Load tenant-specific settings
if (root.has("tenants")) {
JsonNode tenants = root.get("tenants");
tenants.fields().forEachRemaining(tenantEntry -> {
String tenantId = tenantEntry.getKey();
JsonNode tenantConfig = tenantEntry.getValue();
if (tenantConfig.has("settings")) {
JsonNode settings = tenantConfig.get("settings");
Map<String, Object> tenantSettings = new
ConcurrentHashMap<>();
settings.fields().forEachRemaining(settingEntry -> {
String settingName = settingEntry.getKey();
JsonNode value = settingEntry.getValue();
if (!value.isNull()) {
tenantSettings.put(settingName,
parseValue(value));
}
});
tenantSettingsCache.put(tenantId, tenantSettings);
}
});
}
lastModified = fileModified;
lastReloadTime = currentTime;
// Log successful reload
System.err.println(String.format("[CustomSettingProvider] Settings
RELOADED: file=%s, tenants=%d, lastModified=%d, reloadTime=%d",
SETTINGS_FILE_PATH, tenantSettingsCache.size(), lastModified,
lastReloadTime));
} catch (Exception e) {
// Log error but don't fail - use defaults
System.err.println("Error loading tenant settings from " +
SETTINGS_FILE_PATH + ": " + e.getMessage());
}
}
/**
* Map BifroMQ Setting enum names to JSON field names.
* Some settings have different names in the enum vs JSON:
* - MaximumQoS (enum) -> MaxQoS (JSON)
* - MaxUserPayloadBytes (enum) -> MaxPacketSizeBytes (JSON)
*/
private String getJsonFieldName(String settingName) {
// Map Setting enum names to JSON field names
switch (settingName) {
case "MaximumQoS":
return "MaxQoS"; // Enum uses MaximumQoS, JSON uses MaxQoS
case "MaxUserPayloadBytes":
return "MaxPacketSizeBytes"; // BifroMQ enum uses
MaxUserPayloadBytes, JSON uses MaxPacketSizeBytes
case "MaxUserPropertySize":
return "MaxUserPropertySizeBytes";
case "MaxWillMessageSize":
return "MaxWillMessageSizeBytes";
default:
// Other settings use the same name
return settingName;
}
}
/**
* Parse JSON value to appropriate Java type.
* Returns boxed types (Integer, Boolean, Long) as BifroMQ expects boxed
types, not primitives.
*/
private Object parseValue(JsonNode node) {
if (node.isInt()) {
return Integer.valueOf(node.asInt()); // Return boxed Integer, not
primitive int
} else if (node.isBoolean()) {
return Boolean.valueOf(node.asBoolean()); // Return boxed Boolean,
not primitive boolean
} else if (node.isLong()) {
return Long.valueOf(node.asLong()); // Return boxed Long, not
primitive long
} else if (node.isTextual()) {
return node.asText();
}
return null;
}
```
**Key Implementation Details**:
- The `provide()` method is called by BifroMQ's worker threads
- It loads settings from `tenants.json` with 60-second reload interval
- It checks tenant-specific settings first, then defaults, then returns null
for BifroMQ defaults
- It handles type mapping between BifroMQ enum names and JSON field names
- It returns boxed types (Integer, Boolean, Long) as required by BifroMQ
- Enhanced logging proves the method is called and returns correct values
### BifroMQ Behaviour
BifroMQ:
- ✅ Calls the plugin correctly
- ✅ Receives correct values from the plugin
- ❌ Does NOT enforce the provided limits
- ❌ Accepts operations that violate the limits
### Possible Root Causes
1. **Cache Issue**: BifroMQ may be using cached default values instead of
plugin-provided values
2. **Timing Issue**: Settings may be checked at connection time, but not
re-checked at publish/subscribe time
3. **Enforcement Logic Missing**: BifroMQ may not have enforcement logic for
tenant-level settings from the Setting Provider plugin
4. **Configuration Issue**: Some JVM parameter or configuration may be
preventing enforcement
5. **Plugin Integration Issue**: Settings may be retrieved but not used in
enforcement code paths
6. **Default Override**: BifroMQ may be using hardcoded defaults instead of
plugin-provided values
### Testing Methodology
All tests follow the same pattern:
1. Configure a restrictive setting in `tenants.json` for `tenant2`
2. Verify plugin is called and returns correct value (via enhanced logging)
3. Attempt an operation that violates the setting
4. Observe that the operation is accepted instead of rejected
**Test Coverage**: 6 settings tested, 6 settings confirmed NOT enforced
(100% failure rate).
## Requested Action
Please investigate why BifroMQ is not enforcing tenant-level settings
provided by the Setting Provider plugin, even though:
- The plugin is correctly implemented
- The plugin is being called by BifroMQ
- The plugin is returning correct values
- The values are successfully cast and returned
The [official
documentation](https://bifromq.apache.org/docs/plugin/setting_provider/tenantsetting/)
indicates these settings should be enforced, but they are **not being enforced
in practice**.
### Specific Questions
1. Are tenant-level settings from the Setting Provider plugin supposed to be
enforced at publish/subscribe time?
2. Is there a configuration parameter that enables the enforcement of
tenant-level settings?
3. Are there any known issues or limitations with tenant-level settings
enforcement?
4. Should these settings be enforced differently (e.g., at connection time
vs. publish time)?
### Reproduction
The issue can be reproduced by:
1. Setting any tenant-level setting in `tenants.json` (e.g.,
`MaxTopicLevelLength: 10`)
2. Verifying plugin returns the value (check logs)
3. Attempting an operation that violates the setting
4. Observing that the operation is accepted
**Expected**: Operation should be rejected
**Actual**: Operation is accepted
This suggests a fundamental issue with how BifroMQ uses tenant-level
settings from the Setting Provider plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
