Pratyush Bhatt created HDDS-10670:
-------------------------------------
Summary: [Hsync] OverWrite using "sh key put" shouldn't be allowed
on hsynced files.
Key: HDDS-10670
URL: https://issues.apache.org/jira/browse/HDDS-10670
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Pratyush Bhatt
*Scenario:* Do a _sh key put_ on the same key that has been hsynced and is open.
*Observations:*
Open a file, write data and do hsync
{code:java}
Data written on iteration : 1
Data written on iteration : 2
Data written on iteration : 3
Data written on iteration : 4
Data written on iteration : 5
Data written on iteration : 6
Data written on iteration : 7
Data written on iteration : 8
Data written on iteration : 9
Data written on iteration : 10
Hsync completed on file, counter: 10 {code}
----Here the code sleeps of 60 seconds-----
Checking the file is Open and hsynced
{code:java}
ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1151 total open files (est.). Showing 1 open files (limit 100) under path
prefix:
/hsyncvol/hsyncbuck/Client ID Creation time Hsync'ed Open File
Path
112241980463843099 1712676703855 Yes
/hsyncvol/hsyncbuck/-9223372036849133055/File_new_11.txtReached the end of the
list. {code}
Doing a key info till now:
{code:java}
ozone sh key info hsyncvol/hsyncbuck/hsync/File_new_11.txt
24/04/09 15:31:48 INFO protocolPB.OmTransportFactory: Loading OM transport
implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory
as specified by configuration.
24/04/09 15:31:49 INFO client.ClientTrustManager: Loading certificates for
client.
{
"volumeName" : "hsyncvol",
"bucketName" : "hsyncbuck",
"name" : "hsync/File_new_11.txt",
"dataSize" : 51200,
"creationTime" : "2024-04-09T15:31:43.855Z",
"modificationTime" : "2024-04-09T15:31:45.145Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
},
"metadata" : {
"hsyncClientId" : "112241980463843099"
},
"ozoneKeyLocations" : [ {
"containerID" : 2,
"localID" : 113750153625604824,
"length" : 51200,
"offset" : 0,
"keyOffset" : 0
} ],
"file" : true
} {code}
Now OverWrite the same key using _sh key put_
{code:java}
ozone sh key put hsyncvol/hsyncbuck/hsync/File_new_11.txt /etc/passwd
24/04/09 15:32:08 INFO protocolPB.OmTransportFactory: Loading OM transport
implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory
as specified by configuration.
24/04/09 15:32:09 INFO client.ClientTrustManager: Loading certificates for
client.
24/04/09 15:32:10 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
24/04/09 15:32:10 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
24/04/09 15:32:10 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics
system started
24/04/09 15:32:10 INFO metrics.MetricRegistries: Loaded MetricRegistries class
org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl {code}
Key info, successfully overwritten:
{code:java}
ozone sh key info hsyncvol/hsyncbuck/hsync/File_new_11.txt
24/04/09 15:32:17 INFO protocolPB.OmTransportFactory: Loading OM transport
implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory
as specified by configuration.
24/04/09 15:32:17 INFO client.ClientTrustManager: Loading certificates for
client.
{
"volumeName" : "hsyncvol",
"bucketName" : "hsyncbuck",
"name" : "hsync/File_new_11.txt",
"dataSize" : 6644,
"creationTime" : "2024-04-09T15:31:43.855Z",
"modificationTime" : "2024-04-09T15:32:11.283Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
},
"metadata" : { },
"ozoneKeyLocations" : [ {
"containerID" : 1,
"localID" : 113750153625604825,
"length" : 6644,
"offset" : 0,
"keyOffset" : 0
} ],
"file" : true
} {code}
But key is still open in the OpenKeyTable:
{code:java}
ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1151 total open files (est.). Showing 1 open files (limit 100) under path
prefix:
/hsyncvol/hsyncbuck/Client ID Creation time Hsync'ed Open File
Path
112241980463843099 1712676703855 Yes
/hsyncvol/hsyncbuck/-9223372036849133055/File_new_11.txtReached the end of the
list. {code}
Perform LeaseRecovery:
{code:java}
ozone debug recover
--path=ofs://ozone1712556312//hsyncvol/hsyncbuck/hsync/File_new_11.txt
24/04/09 15:32:32 INFO protocolPB.OmTransportFactory: Loading OM transport
implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory
as specified by configuration.
24/04/09 15:32:32 INFO client.ClientTrustManager: Loading certificates for
client.
Lease recovery SUCCEEDED on
ofs://ozone1712556312//hsyncvol/hsyncbuck/hsync/File_new_11.txt {code}
Still the key is open:
{code:java}
ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1151 total open files (est.). Showing 1 open files (limit 100) under path
prefix:
/hsyncvol/hsyncbuck/Client ID Creation time Hsync'ed Open File
Path
112241980463843099 1712676703855 Yes
/hsyncvol/hsyncbuck/-9223372036849133055/File_new_11.txtReached the end of the
list. {code}
-- 60 seconds of code pause is resumed now--
Still able to write the data into the file and hsync
{code:java}
Data written on iteration : 11
Data written on iteration : 12
Data written on iteration : 13
Data written on iteration : 14
Data written on iteration : 15
Data written on iteration : 16
Data written on iteration : 17
Data written on iteration : 18
Data written on iteration : 19
Data written on iteration : 20
Hsync completed on file, counter: 20 {code}
Basically looks like lease recovery gets broken if we allow such overwriting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
