Re: [PR] KAFKA-15388: Handling remote segment read in case of log compaction [kafka]

[via GitHub]

iit2009060 commented on PR #15060:
URL: https://github.com/apache/kafka/pull/15060#issuecomment-1879980873

   > @satishd @divijvaidya @kamalcph I am able to reproduce the above scenario 
using retention feature instead of log compaction.
   > 
   > The overall problem is we are sending MemoryRecords.Empty when unable to 
find offset even though active segments can still have the data.
   > 
   > Consider the below scenario
   > 
   > 1. create topic test8 with partition 0 and with remote storage enabled.
   > 2. Current status of topic (Offset 0,2,3,4)
   > 
   > <img alt="Screenshot 2024-01-05 at 12 40 13 AM" width="992" 
src="https://private-user-images.githubusercontent.com/59436466/294679830-ab83f9e3-a44b-4c22-8d72-20a07fc6ce6d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2Nzk4MzAtYWI4M2Y5ZTMtYTQ0Yi00YzIyLThkNzItMjBhMDdmYzZjZTZkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwNzlkNzQ1NTRmYzFmZjU5OTNjYmMwZDM5OTU5YmZjNjBhZjczMzdjNjg4ZDQxZWI2NzdiM2U5ZTlhNzQwMjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.M4Y1cvGSDNozMQTsAWwwq9K8ZHkCnGUZLqy6fKBk5GY";>
 3. When we make a fetch request with offset 1 <img alt="Screenshot 2024-01-06 
at 5 19 01 PM" width="984" sr
 
c="https://private-user-images.githubusercontent.com/59436466/294679851-9ef58907-a545-403b-a004-20e094945d30.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2Nzk4NTEtOWVmNTg5MDctYTU0NS00MDNiLWEwMDQtMjBlMDk0OTQ1ZDMwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNmNzY2OTUzMTUwMGE3NzNhODE5ZmJlYWIxYWRjNDBhNjRjMTJlNWUzNTllMTQ5MDFhMmZmMDdiM2E0MDAxNTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.HeHVo2nfLE0HLDe4QkEcrFzx1_hUGrwgnaifTzJPo_I";>
 4. To delete remote segments I have set up the configuration retention.ms=1000 
<img alt="Screenshot 2024-01-06 at 5 21 24 PM" width="988" 
src="https://private-user-images.gi
 
thubusercontent.com/59436466/294679912-ba5b34cd-b9f1-4bb3-9574-8ebe9cd37741.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2Nzk5MTItYmE1YjM0Y2QtYjlmMS00YmIzLTk1NzQtOGViZTljZDM3NzQxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUxZjRmODg4MjRiNjE3Njc5M2E1Y2IyYzViZWVlMzdhNTU2Mzk4MzMzMmI0ZjcxMGZiZDUyYjZjYmRhY2EyZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.loKBLUJmgPtGvrnAFspaabHWZDzMPl_c1M8K4mqdRZQ">
 5. Now once the data is deleted, reset the local and remote retention to 1 
hour. Produce some data <img alt="Screenshot 2024-01-06 at 5 22 59 PM" 
width="995" src="https://private-user-images.githubuserconte
 
nt.com/59436466/294680022-ee91b9a7-8fbe-4586-8ca8-3164fe82bddd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2ODAwMjItZWU5MWI5YTctOGZiZS00NTg2LThjYTgtMzE2NGZlODJiZGRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA0NDI1MjRiM2YwNWRkNGJiYjE2NWVmNzhiOTBhNjEzYmU5YTFjNTJmYTQ4N2Q1NjgyYjQwMTE1ZmY4MDYwYTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.OWvD41CLgnMU69aL_4z5Lak-KlZKCnxRl7apaDQx-DA">
 7. When I am trying to fetch offset 1 from the topic test 8 partition 0 , it 
never able to respond. Ideally it should pick the data from the active segment 
whose offset starts at 6. <img alt="Screenshot 2024-01-06 at 5 
 24 44 PM" width="969" 
src="https://private-user-images.githubusercontent.com/59436466/294680083-15a560ca-9396-4a73-9c16-8812cda36adf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2ODAwODMtMTVhNTYwY2EtOTM5Ni00YTczLTljMTYtODgxMmNkYTM2YWRmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwZGE4MWRiNGViNTQzZDliNjhlNmVmZDFlOGQxY2NhOWI3ZjlkZWYxZmQ4MGU2ODk1ZWRlZTUyOTNmNzNjNjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Pe2rQNuiGLQE3hdm7KaUGSILVFf_FwNz3rZ2fW_1BkA";>
 8. It happened because requestedoffset 1 is not greater than 
log.logStartOffset which is 6 <img alt="Screenshot 2024-01-06 at 5 27 42 PM" 
width="1791
 " 
src="https://private-user-images.githubusercontent.com/59436466/294680250-067ed7d9-5bc7-4d56-b0c9-799dabdad9fa.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2ODAyNTAtMDY3ZWQ3ZDktNWJjNy00ZDU2LWIwYzktNzk5ZGFiZGFkOWZhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM3NWZlZjEwMjRiZTVlNTA4NmMxMzk1MDllMGM1YjE5MTQ0OWU0YmE0ODcwZjBlN2MwMWM2YzEzMzk4ZGY0ZDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.7Steo5pil5JVVFQIakJjxBNvVLwnJKREFaJohAbB5bc";>
   > 9. We create Empty Record Response.
   > 
   > <img alt="Screenshot 2024-01-06 at 5 26 04 PM" width="1760" 
src="https://private-user-images.githubusercontent.com/59436466/294680642-f93a53e7-f7ff-4885-849b-f95fedd830e9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ2MTI0MjksIm5iZiI6MTcwNDYxMjEyOSwicGF0aCI6Ii81OTQzNjQ2Ni8yOTQ2ODA2NDItZjkzYTUzZTctZjdmZi00ODg1LTg0OWItZjk1ZmVkZDgzMGU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA3VDA3MjIwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNhNmJkYjRlMjc3YjU4MmIyMGY5MjA3NWEwYjFiMTQyOWNiNzY0MzYyNTIzMjg4NTQ5Yjk1OGVjMDA3OTVhMzImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.2QCkvA_cekLqN_7adPhfu9hoWjiAy4SgHVY7hXNB5uw";>
   > In general this is happening because The next fetch request offset 
increments only when there is a record list. But the Remote fetch 
implementation by default if not able to find the records sents Empty Records.
   > 
   > We should create separate JIRA to track this .The issue is not specific to 
log compaction but it will happen whenever we sent Records Empty from remote 
storage.
   
   @satishd  @divijvaidya  I have created separate ticket to track the 
scenarios when RemoteFetch return Empty Records.
   https://issues.apache.org/jira/browse/KAFKA-16088


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org