Yes, those are potential issues if you make your Kafka cluster publicly accessible. There are some features currently being worked on that could help address these problems (some security on connections and quotas), but they are still in progress. You'll probably want a proxy layer to handle this.
As for auto-scaling, you can do this but you should be aware that Kafka doesn't currently do automatic balancing of data as brokers are added. You'll need to manage that process yourself when new brokers are added. -Ewen On Thu, Jul 16, 2015 at 7:35 PM, Chandrashekhar Kotekar < shekhar.kote...@gmail.com> wrote: > Thanks a lot Ewen and Edward for your valuable answers. According to new > update from admin side, they can allow TCP only connections on Kafka > brokers. > > Now another problem is that we want to keep Kafka brokers in AWS so that > kafka brokers can be auto scaled in/out. As most of instances in AWS do not > have public IP addresses, first we have to assign public IP to all Kafka > brokers and keep them visible over internet. > > I would like to know if there will be any security issue like DoS attack or > malicious user sending Kafka messages or something like that? > > > Thanks, > Chandrash3khar Kotekar > Mobile - +91 8600011455 > > On Fri, Jul 17, 2015 at 4:57 AM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Chandrashekhar, > > > > If the firewall rules allow any TCP connection on those ports, you can > just > > use Kafka directly and change the default port. If they actually verify > > that its HTTP traffic then you'd have to the REST Proxy Edward mentioned > or > > another HTTP-based proxy. > > > > -Ewen > > > > On Thu, Jul 16, 2015 at 9:23 AM, Edward Ribeiro < > edward.ribe...@gmail.com> > > wrote: > > > > > Maybe what you are looking for is Kafka REST Proxy: > > > http://docs.confluent.io/1.0/kafka-rest/docs/intro.html > > > > > > Edward > > > > > > On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar < > > > shekhar.kote...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > In my project Kafka producers won't be in the same network of Kafka > > > brokers > > > > and due to security reasons other ports are blocked. > > > > > > > > I would like to know if it is possible to run Kafka brokers on HTTP > > port > > > > (8080) so that Kafka producers will send Kafka messages over HTTP and > > > > brokers can store them until consumers consume them. > > > > > > > > I tried to search for this type of question in mailing list but > > couldn't > > > > find exact question/answer. Sorry if this is duplicate question. > > > > > > > > Thanks, > > > > Chandrash3khar Kotekar > > > > Mobile - +91 8600011455 > > > > > > > > > > > > > > > -- > > Thanks, > > Ewen > > > -- Thanks, Ewen
