Re: [Spark-Core]port opened by the SparkDriver is vulnerable to flooding attacks

2018-02-28 Thread Marcelo Vanzin 
The mechanism to close idle connections already exists. It doesn't
mean you can just use it as is in existing connections. So if you want
to go and fix that, you're going to have to figure out that part. Or
figure out a different solution.

Either way, file a bug so that this is properly tracked. If you want
to try to fix it, open a PR with the changes.

On Wed, Feb 28, 2018 at 9:42 AM, Sandeep Katta
 wrote:
> Yeh monitor is present but for some cases like long running job I found App
> master is idle.so it will end up closing the App master’s channel so job
> will not be completed.
>
> So needed a mechanism to close only invalid connections .
>
>
> On Wed, 28 Feb 2018 at 10:54 PM, Marcelo Vanzin  wrote:
>>
>> Spark already has code to monitor idle connections and close them.
>> That's in TransportChannelHandler.java.
>>
>> If there's anything to do here, it's to allow all users of the
>> transport library to support the "close idle connections" feature of
>> that class.
>>
>> On Wed, Feb 28, 2018 at 9:07 AM, sandeep_katta
>>  wrote:
>> > In case of client mode App Master and Driver are in different JVM
>> > process,the
>> > port opened by the Driver is vulnerable for flooding attacks as it is
>> > not
>> > closing the IDLE connections.
>> >
>> > I am thinking to fix this issue using below mechanism
>> >
>> > 1.Expose configuration to close the IDLE connections as secondary
>> > developers
>> > can decide based on their use case
>> >
>> > 2.Maintain the SET of validConnections,the condition for
>> > validConnections is
>> > *TransportChannelHandler.channelRead* event will be fired for the valid
>> > connections.
>> >
>> > 3.Whenever there is timeout(no communication happened across the channel
>> > for
>> > x time),close the connection.
>> >
>> >if !validConnectionSet then
>> >   close the connection
>> >
>> > Note: Timeout value is configurable using the config
>> > "spark.network.timeout"
>> >
>> >
>> > can this solution be used to solve this issue,if any suggestions are
>> > also
>> > welcomed.
>> >
>> > Regards
>> > Sandeep
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>>
>>
>> --
>> Marcelo



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Spark-Core]port opened by the SparkDriver is vulnerable to flooding attacks

2018-02-28 Thread Sandeep Katta 
Yeh monitor is present but for some cases like long running job I found App
master is idle.so it will end up closing the App master’s channel so job
will not be completed.

So needed a mechanism to close only invalid connections .


On Wed, 28 Feb 2018 at 10:54 PM, Marcelo Vanzin  wrote:

> Spark already has code to monitor idle connections and close them.
> That's in TransportChannelHandler.java.
>
> If there's anything to do here, it's to allow all users of the
> transport library to support the "close idle connections" feature of
> that class.
>
> On Wed, Feb 28, 2018 at 9:07 AM, sandeep_katta
>  wrote:
> > In case of client mode App Master and Driver are in different JVM
> process,the
> > port opened by the Driver is vulnerable for flooding attacks as it is not
> > closing the IDLE connections.
> >
> > I am thinking to fix this issue using below mechanism
> >
> > 1.Expose configuration to close the IDLE connections as secondary
> developers
> > can decide based on their use case
> >
> > 2.Maintain the SET of validConnections,the condition for
> validConnections is
> > *TransportChannelHandler.channelRead* event will be fired for the valid
> > connections.
> >
> > 3.Whenever there is timeout(no communication happened across the channel
> for
> > x time),close the connection.
> >
> >if !validConnectionSet then
> >   close the connection
> >
> > Note: Timeout value is configurable using the config
> "spark.network.timeout"
> >
> >
> > can this solution be used to solve this issue,if any suggestions are also
> > welcomed.
> >
> > Regards
> > Sandeep
> >
> >
> >
> > --
> > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
>
>
> --
> Marcelo
>

Re: [Spark-Core]port opened by the SparkDriver is vulnerable to flooding attacks

2018-02-28 Thread Marcelo Vanzin 
Spark already has code to monitor idle connections and close them.
That's in TransportChannelHandler.java.

If there's anything to do here, it's to allow all users of the
transport library to support the "close idle connections" feature of
that class.

On Wed, Feb 28, 2018 at 9:07 AM, sandeep_katta
 wrote:
> In case of client mode App Master and Driver are in different JVM process,the
> port opened by the Driver is vulnerable for flooding attacks as it is not
> closing the IDLE connections.
>
> I am thinking to fix this issue using below mechanism
>
> 1.Expose configuration to close the IDLE connections as secondary developers
> can decide based on their use case
>
> 2.Maintain the SET of validConnections,the condition for validConnections is
> *TransportChannelHandler.channelRead* event will be fired for the valid
> connections.
>
> 3.Whenever there is timeout(no communication happened across the channel for
> x time),close the connection.
>
>if !validConnectionSet then
>   close the connection
>
> Note: Timeout value is configurable using the config "spark.network.timeout"
>
>
> can this solution be used to solve this issue,if any suggestions are also
> welcomed.
>
> Regards
> Sandeep
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>



-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org