Today following a re-balance operation in our 0.8.0 cluster , we had a bunch of disks fill up, even though our retention property values are designed to prevent that level of disk usage.
When we investigated, it appeared the following had occurred: 1. Broker A was in the replica list for partitions 1,2,3 2. We issued a re-balance per the re-assign partition tool - https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool 3. Wait for the the new replica / leader to join the ISR 4. In this case, Broker A was neither a leader nor replica for any of partitions 1,2,3 after the re-balance, yet the data files for those partitions remained on Broker A's data folder after the re-balance had completed. Is this expected behavior? Are the segments remaining on Broker A, but no longer used by the cluster going to lie around forever? Are they subject only to the time retention policy from that point on?
