Gabriel39 commented on code in PR #70697:
URL: https://github.com/apache/doris/pull/70697#discussion_r4610840523
##########
fe/fe-core/src/main/java/org/apache/doris/common/OssThriftClientTransportProvider.java:
##########
@@ -18,16 +18,39 @@
package org.apache.doris.common;
import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.tls.PrivateRpcTls;
+import org.apache.doris.tls.TlsThriftSocket;
+import org.apache.doris.tls.server.TlsProtocolSet.Protocol;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransport;
import org.apache.thrift.transport.layered.TFramedTransport;
public class OssThriftClientTransportProvider implements
ThriftClientTransportProvider {
+ @Override
+ public boolean supportsPrivateFileKeys() {
+ // Legacy subclasses may override transport creation without using the
OSS TLS implementation.
+ return getClass() == OssThriftClientTransportProvider.class
+ && PrivateRpcTls.supportsPrivateFileKeys(Protocol.THRIFT);
+ }
+
@Override
public TTransport createTransport(TNetworkAddress address, int timeoutMs,
boolean nonBlocking) throws Exception {
- return nonBlocking
- ? new TFramedTransport(new TSocket(address.hostname,
address.port, timeoutMs))
+ TSocket socket = PrivateRpcTls.enabled(Protocol.THRIFT)
+ ? new TlsThriftSocket(address.hostname, address.port,
timeoutMs)
Review Comment:
Integrated the TLS prerequisite fixes in e3a7782e1553. Heartbeat pools
retain the registered DNS hostname separately from the resolved connection/pool
address, including authenticated reopen and address turnover. Both Java and
native clients verify the logical peer identity. The selected FE tests include
the real-listener trust/identity, reopen and DNS turnover coverage; all 145
selected FE tests pass.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergEncryptionRead.java:
##########
@@ -60,10 +91,18 @@ static void check(Table table, IcebergCatalogProperties
catalog, ConnectorContex
throw failure("UNSUPPORTED_ENCRYPTION_MANAGER", "Encrypted Iceberg
reads require the SDK standard "
+ "encryption scheme with native Parquet AES_GCM_V1
files");
}
- // A PME-enabled Arrow build does not prove key-transport,
delete-reader or mixed-version scheduling
- // support. Until explicit BE negotiation is wired, even an enabled FE
must never emit encrypted tasks.
- throw failure("BACKEND_CAPABILITY_UNAVAILABLE", "Encrypted Iceberg
read capability is not verified "
- + "by the engine; native encrypted scan execution is not
available");
+ // Do not initialize a KMS before the engine verifies both protected
transports and native readers.
+ if (context == null ||
context.getStorageContext().getNativeFileEncryptionVersion() < 1) {
+ throw failure("BACKEND_CAPABILITY_UNAVAILABLE", "Encrypted Iceberg
read capability is not verified "
+ + "by the engine; native encrypted scan execution is not
available");
+ }
+ if (manager == null) {
+ manager = table.encryption();
+ }
+ if (!(manager instanceof StandardEncryptionManager)) {
Review Comment:
Fixed in e3a7782e1553. Read admission no longer determines write
eligibility. Native write planning, transaction begin, SDK refresh and final
metadata CAS independently reject encrypted targets with WRITE_UNAVAILABLE. The
existing validating operations retain schema/identity/snapshot checks,
authentication and OCC behavior without adding an unconditional metadata-file
reread for ordinary writes. Real SDK tests cover every WriteOperation,
concurrent encryption adoption at refresh/final CAS and a successful plaintext
commit. The SQL regression also proves encrypted INSERT/OVERWRITE refusal,
unchanged encrypted rows and successful plaintext INSERT.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java:
##########
@@ -274,16 +278,20 @@ static final class TableOwner {
private final String snapshotMetadataJson;
private final String snapshotMetadataLocation;
final MetaCacheSizeEstimate sizeEstimate;
+ private final boolean cacheable;
// A newly loaded value starts with a cache reference and a temporary
loader reference. The temporary
// reference bridges publication/discard to the first borrow,
including invalidation-before-publication.
private final AtomicInteger references;
private TableOwner(Table table, Runnable cleanup, boolean loading,
boolean estimateWeight) {
this.table = table;
this.cleanup = cleanup;
+ // SDK managers cache unwrapped keys. Even an unweighted cache
must release this table
+ // with its statement, so a later statement performs its own
catalog/KMS authorization.
+ this.cacheable = !IcebergEncryptionRead.isEncrypted(table);
Review Comment:
Fixed in e3a7782e1553. Encrypted tables explicitly throw
SQL_CACHE_UNAVAILABLE from the connector version provider, so both admission
and replay use the existing refusal path rather than the legacy fallback for an
empty token. Tests cover rejection before key unwrap and eviction of a
previously warm entry. The enabled SQL regression sets both cache switches,
proves a plaintext PhysicalSqlCache hit, warms encrypted COUNT/SUM, revokes KMS
permission at the same endpoint, verifies both queries fail repeatedly, then
verifies recovery. The plaintext cache-hit control remains valid during
revocation. Expected SQL output was generated through the repository runner and
replayed successfully.
##########
fe/fe-core/src/main/java/org/apache/doris/system/Backend.java:
##########
@@ -886,7 +899,14 @@ public String toString() {
* return true if any port changed, or alive state is changed.
*/
public boolean handleHbResponse(BackendHbResponse hbResponse, boolean
isReplay) {
- boolean isChanged = false;
+ FileEncryptionCapability previousCapability = fileEncryptionCapability;
+ fileEncryptionCapability = hbResponse.getStatus() == HbStatus.OK
+ && hbResponse.getNativeFileEncryptionVersion() > 0
+ ? new
FileEncryptionCapability(hbResponse.getNativeFileEncryptionVersion(),
hbResponse.getHbTime())
+ : null;
+ // Positive leases must reach followers every round. Missing/failed
heartbeats must revoke them
+ // immediately, even while the usual liveness grace still considers
the backend alive.
+ boolean isChanged = previousCapability != null ||
fileEncryptionCapability != null;
Review Comment:
Fixed in e3a7782e1553. The leader processes and retains the original BAD
response for failure counting, eviction and diagnostics. When capability
revocation changes state before the liveness threshold, the journal instead
carries an OK snapshot of the resulting cached backend state with capability
zero; older followers can interpret it without prematurely marking the backend
dead. The serialized HbPackage test uses tolerance three and checks immediate
capability loss, matching leader/follower liveness through each failure,
preserved journal fields and successful recovery. It fails on the original
premature-death behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]