Hello M.Manna,
Do you mean we should not use Kafka in windows production until the disk I/O
issues(For which you have given workaround) are resolved officilally?
ThanksHarish
On Monday, July 17, 2017, 5:31:11 PM GMT+5:30, M. Manna <manme...@gmail.com>
wrote:
The workaround is to download the sources and create a wrapper around method
forceUnmap in LogSegment.scala file. Please do not take it a solution as no one
has approved this yet on KAFKA-1194 ticket. but it works.
I tried to update the scala sources to run the unit tests - but it fails all
the time. Howerver, the fix works. Also, the log files are actually okay, it's
the index and timeIndex files which throws RTE.
Do the following (Prerequisite - you have to follow the src download and setup
instruction from Kafka website - Also use 0.10.2.1 or higher):
1) Create a new wrapper in AbstractIndex.scala directly under def forceUnmap
// Forces unmapping of multibyte map files on Windowsdef forceUnmapOnWindows()
{if (OperatingSystem.IS_WINDOWS) forceUnmap(mmap)}
2. Find the method changeFileSuffixes and invoke the above method (i.e.
forceUnmapOnWindows() ) for index and timeIndex files - just before:
i) The try catch block with index file ii) The try catch block with
timeIndex file.
So it will become the following:
def changeFileSuffixes(oldSuffix: String, newSuffix: String) {
def kafkaStorageException(fileType: String, e: IOException) =
new KafkaStorageException(s"Failed to change the $fileType file suffix
from $oldSuffix to $newSuffix for log segment $baseOffset", e)
try log.renameTo(new File(CoreUtils.replaceSuffix(log.file.getPath,
oldSuffix, newSuffix)))
catch {
case e: IOException => throw kafkaStorageException("log", e)
}
index.forceUnmapOnWindows()
try index.renameTo(new File(CoreUtils.replaceSuffix(index.file.getPath,
oldSuffix, newSuffix)))
catch {
case e: IOException => throw kafkaStorageException("index", e)
}
timeIndex.forceUnmapOnWindows()
try timeIndex.renameTo(new
File(CoreUtils.replaceSuffix(timeIndex.file.getPath, oldSuffix, newSuffix)))
catch {
case e: IOException => throw kafkaStorageException("timeindex", e)
}
}
3. Execute gradlew -jar --debug (REMEMBER - you need to setup everything with
the project beforehand, otherwise it won't work).
Once the build is complete - get the core jar and dump it in your kafka broker
lib folder (it will replace the old one).
4. Redeploy and check if the cleanup is working.
KR,
On 17 July 2017 at 12:21, harish jadhav <harishjadhav1...@yahoo.com> wrote:
Hello M.Manna,
Thanks for your quick and valuable response.
Actually I am getting following error -
Caused by: java.nio.file. FileSystemException: \tmp\kafka-logs\TopicName-2\
00000000000000007262.log-> \tmp\kafka-logs\TopicName
-2\00000000000000007262.log. deleted: Theprocess cannot access the file because
it is being used by another process.
Uncaught exception in scheduled task
'kafka-log-retention'(kafka.utils.KafkaScheduler)
kafka.common. KafkaStorageException: Failed to change the log file suffixfrom
to .deleted for log segment 7262
There must be some workaround available for above issues. However I don't know
what other potential issues can come in future. Hence need your suggestion on
whether to continue to use Kafka in windows production system.
Please guide us on this.
ThanksHarish
On Monday, July 17, 2017, 2:15:15 PM GMT+5:30, M. Manna <manme...@gmail.com>
wrote:
Hi,
First I apologise and should just correct myself. I should use "Disk/File level
operations" instead of using "I/O" since that refers to different things.
What I mean to say that file deletion/renaming doesn't work even when you
provide full permissions in Windows. But Unix/Linux based system it's okay.
This can be called as "Deletion/Renaming of shared files" i.e. which are
already opened for appending/editing.
As an example try to run the uni tests for Kafka on Windows where file
renaming/deletion occurs as part of the tests.They will fail. But same process
works all good on Linux.
>From the official documentation, Kafka has already made a disclaimer that
>Windows Implementation is not fully vetted (in progress). I believe all the
>top examples from LinkedIn are based on Unix/Linux. You are actually okay to
>do PoC activities on windows - log retention/rename/deletion will work fine.
I hope this helps you understand.
KR,
On 17 July 2017 at 09:22, harish jadhav <harishjadhav1...@yahoo.com> wrote:
Hi M.Manna,
Thanks for the update.
If possible, can you please share which admin I/O level functionality will
impact? Do you mean something related to log files renaming?
In documentation, there is no pointer with respect to I/O.
Kindly advice.
ThanksHarish
On Saturday, July 15, 2017, 1:59:58 PM GMT+5:30, M. Manna <manme...@gmail.com>
wrote:
Kafka has some issues wuth IO level admin functionalities e.g. file
deletion/renaming doesn't work out of the box like Linux. However, user base is
growing and there's a momentum to support Windows OS seamlessly like Linux.
I have been running some stress tests on Windows and the performance is quite
consistent. If you have any findings you would like to share, please publish
them.
Please read Kafka documentation where there are some disclaimers regarding OS
support.
On 15 Jul 2017 1:27 am, "Eno Thereska" <eno.there...@gmail.com> wrote:
Hi Harish,
I believe many people/orgs use it on Windows. We rely on the community to
test/fix/answer any Windows questions, same as with Linux or MacOS. However,
based on what I've observed, perhaps there are more people answering
Linux-related questions.
Eno
> On 14 Jul 2017, at 13:24, harish jadhav <harishjadhav1...@yahoo.com. INVALID>
> wrote:
>
> Hello Team,
>
> I am exploring Apache Kafka and found that one of the best MQ I have
> encountered. I was exploring option to use it in Windows machine and started
> some kind of proof of concept work referring installation section on windows
> and it work perfectly. Later realized that Kafka documentation says under
> Hardware and OS section " We have seen a few issues running on Windows and
> Windows is not currently a well supported platform though we would be happy
> to change that. "
>
> I am curios to know is there any actual issue running Kafka in windows OS as
> whichever feature I am using with single instance ( Producer-Consumer) works
> perfectly in windows test bed. Can I use it production windows machines?
>
> Please advice.
>
> Thanks
> Harish
