Hi KR,
Thanks for clarifyication and thank you for spending your valuable time to
assist on our issues.
Our requirement is to have Kafka in Windows environment. The messages in Kafka
gets pushed from external entities; we have consumers which consume those
records and insert to mySQL database. Also we want to keep the messages on
Kafka for maximum one week for making sure we won't loose any data when
connectivity is not there to DB.
We don't use Linux as our setups will be purely windows house (not even Linux
dockers or VM ).
Indeed we have not faced any other issues apart from the file deletion/renaming
issue which I reported to you in my previous mail. As you suggested, we can
cleanup the logs from sepearte utility.
My only concern is that - as document suggests running kafka on windows has
some issues hence not able to decide on using it on windows production bed.
Thanks
Harish
On Mon, 7/17/17, M. Manna <manme...@gmail.com> wrote:
Subject: Re: Windows OS platform support
To: "harish jadhav" <harishjadhav1...@yahoo.com>
Cc: "Kafka Users" <users@kafka.apache.org>
Date: Monday, July 17, 2017, 4:04 PM
I have
never said that that we shouldn't use Kafka officially
for Windows Production Environment. I am running a major PoC
which involves developing solution on Windows 10 - but my
production docker containers are using Alpine Linux so I
won't have any issues for the final integration
pipeline.
What I would like
to clarify is that if you are going for a full Windows
pipeline for your implementation (i.e. dev+delivery both on
Windows platoform) - you might want to do more research.
This user group is awesome and everyone is very active in
getting all the issues resolved asap.
We have seen good performance on
Windows and they are more or less consistent with all the
metrics published in Kafka - if you don't have a strict
requirement to do log cleanup you can use the workaround I
mentioned, or just clean the logs yourself.
What is your implementation
requirement, exactly? Do you need to preserve all the
messages (i.e. ability to replay)? Or would you be happy to
delete them during a scheduled maintenance window
?
KR,
On 17 July 2017 at 16:19,
harish jadhav <harishjadhav1...@yahoo.com>
wrote:
Hello
M.Manna,
Do you mean
we should not use Kafka in windows production until the disk
I/O issues(For which you have given workaround) are resolved
officilally?
ThanksHarish
On Monday, July 17, 2017,
5:31:11 PM GMT+5:30, M. Manna <manme...@gmail.com>
wrote:
The workaround is to
download the sources and create a wrapper around method
forceUnmap in LogSegment.scala file. Please do not take it a
solution as no one has approved this yet on KAFKA-1194
ticket. but it works.
I
tried to update the scala sources to run the unit tests -
but it fails all the time. Howerver, the fix works. Also,
the log files are actually okay, it's the index and
timeIndex files which throws RTE.
Do the following (Prerequisite - you
have to follow the src download and setup instruction from
Kafka website - Also use 0.10.2.1 or higher):
1) Create a new wrapper in
AbstractIndex.scala directly under def
forceUnmap
//
Forces unmapping of multibyte map files on
Windowsdef forceUnmapOnWindows() {if
(OperatingSystem.IS_WINDOWS)
forceUnmap(mmap)}
2. Find the method
changeFileSuffixes and invoke the above method (i.e.
forceUnmapOnWindows() ) for index and timeIndex files -
just before:
i) The try catch block with index file
ii) The try catch block with timeIndex file.
So it will become the
following:
def
changeFileSuffixes(oldSuffix: String, newSuffix: String)
{
def kafkaStorageException( fileType:
String, e: IOException) =
new
KafkaStorageException(s"Failed to change the $fileType
file suffix from $oldSuffix to $newSuffix for log segment
$baseOffset", e)
try
log.renameTo(new File(CoreUtils.replaceSuffix(
log.file.getPath, oldSuffix, newSuffix)))
catch {
case e: IOException
=> throw kafkaStorageException("log", e)
}
index.forceUnmapOnWindows()
try
index.renameTo(new File(CoreUtils.replaceSuffix(
index.file.getPath, oldSuffix, newSuffix)))
catch {
case e:
IOException => throw
kafkaStorageException("index", e)
}
timeIndex.forceUnmapOnWindows( )
try
timeIndex.renameTo(new File(CoreUtils.replaceSuffix(
timeIndex.file.getPath, oldSuffix, newSuffix)))
catch {
case e:
IOException => throw kafkaStorageException("
timeindex", e)
}
} 3.
Execute gradlew -jar --debug (REMEMBER - you need to setup
everything with the project beforehand, otherwise it
won't work).
Once
the build is complete - get the core jar and dump it in your
kafka broker lib folder (it will replace the old
one).
4. Redeploy
and check if the cleanup is working.
KR,
On 17 July 2017 at 12:21, harish
jadhav <harishjadhav1...@yahoo.com>
wrote:
Hello M.Manna,
Thanks
for your quick and valuable response.
Actually I am getting following
error -
Caused by:
java.nio.file. FileSystemException:
\tmp\kafka-logs\TopicName-2\
00000000000000007262.log
-> \tmp\kafka-logs\TopicName
-2\00000000000000007262.log. deleted: The
process cannot access the file because it is being used by
another process.
Uncaught exception in
scheduled task 'kafka-log-retention'
(kafka.utils.KafkaScheduler)
kafka.common.
KafkaStorageException: Failed to change the log file suffix
from to .deleted for log segment 7262
There must be
some workaround available for above issues. However I
don't know what other potential issues can come in
future. Hence need your suggestion on whether to continue to
use Kafka in windows production system.
Please guide us on
this.
ThanksHarish
On Monday,
July 17, 2017, 2:15:15 PM GMT+5:30, M. Manna <manme...@gmail.com>
wrote:
Hi,
First I
apologise and should just correct myself. I should use
"Disk/File level operations" instead of using
"I/O" since that refers to different
things.
What I mean to
say that file deletion/renaming doesn't work even when
you provide full permissions in Windows. But Unix/Linux
based system it's okay. This can be called as
"Deletion/Renaming of shared files" i.e. which are
already opened for appending/editing.
As an example try to run the uni
tests for Kafka on Windows where file renaming/deletion
occurs as part of the tests.They will fail. But same process
works all good on Linux.
From the official documentation,
Kafka has already made a disclaimer that Windows
Implementation is not fully vetted (in progress). I believe
all the top examples from LinkedIn are based on Unix/Linux.
You are actually okay to do PoC activities on windows - log
retention/rename/deletion will work fine.
I hope this helps you
understand.
KR,
On
17 July 2017 at 09:22, harish jadhav <harishjadhav1...@yahoo.com>
wrote:
Hi
M.Manna,
Thanks for the update.
If possible, can you
please share which admin I/O level functionality will
impact? Do you mean something related to log files
renaming?
In
documentation, there is no pointer with respect to
I/O.
Kindly advice.
ThanksHarish
On
Saturday, July 15, 2017, 1:59:58 PM GMT+5:30, M. Manna
<manme...@gmail.com>
wrote:
Kafka
has some issues wuth IO level admin functionalities e.g.
file deletion/renaming doesn't work out of the box like
Linux. However, user base is growing and there's a
momentum to support Windows OS seamlessly like
Linux.
I have been running
some stress tests on Windows and the performance is quite
consistent. If you have any findings you would like to
share, please publish them.
Please read Kafka documentation
where there are some disclaimers regarding OS
support.
On
15 Jul 2017 1:27 am, "Eno Thereska" <eno.there...@gmail.com>
wrote:
Hi
Harish,
I believe many people/orgs use it on Windows. We rely on the
community to test/fix/answer any Windows questions, same as
with Linux or MacOS. However, based on what I've
observed, perhaps there are more people answering
Linux-related questions.
Eno
> On 14 Jul 2017, at 13:24, harish jadhav <harishjadhav1...@yahoo.com.
INVALID> wrote:
>
> Hello Team,
>
> I am exploring Apache Kafka and found that one of the
best MQ I have encountered. I was exploring option to use it
in Windows machine and started some kind of proof of concept
work referring installation section on windows and it work
perfectly. Later realized that Kafka documentation says
under Hardware and OS section " We have seen a few
issues running on Windows and Windows is not currently a
well supported platform though we would be happy to change
that. "
>
> I am curios to know is there any actual issue running
Kafka in windows OS as whichever feature I am using with
single instance ( Producer-Consumer) works perfectly in
windows test bed. Can I use it production windows
machines?
>
> Please advice.
>
> Thanks
> Harish
