Re: Use of Accumulators

2017-11-14 Thread Kedarnath Dixit 

Yes!



Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.


From: Holden Karau 
Sent: 14 November 2017 20:04:50
To: Kedarnath Dixit
Cc: user@spark.apache.org
Subject: Re: Use of Accumulators

And where do you want to read the toggle back from? On the driver?

On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit 
mailto:kedarnath_di...@persistent.com>> wrote:

Hi,



Inside the transformation if there is any change for the Variable’s associated 
data, I want to just toggle it saying there is some change while processing the 
data.



Please let me know if we can runtime do this.





Thanks!

~Kedar Dixit

Bigdata Analytics at Persistent Systems Ltd.



From: Holden Karau [via Apache Spark User List] 
[mailto:ml+s1001560n29995...@n3.nabble.com]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit 
mailto:kedarnath_di...@persistent.com>>

Subject: Re: Use of Accumulators



So you want to set an accumulator to 1 after a transformation has fully 
completed? Or what exactly do you want to do?



On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden 
email]<http:///user/SendEmail.jtp?type=node&node=29995&i=0>> wrote:

Confirmed ,you can use Accumulators :)



Regards,

Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden 
email]<http:///user/SendEmail.jtp?type=node&node=29995&i=1>> wrote:

Hi,



We need some way to toggle the flag of  a variable in transformation.



We are thinking to make use of spark  Accumulators for this purpose.



Can we use these as below:



Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0



In one of the transformations if we need to make Variable2's value to 1. Can we 
achieve this using Accumulators? Please confirm.



Thanks!



With Regards,

~Kedar Dixit



[hidden email]<http:///user/SendEmail.jtp?type=node&node=29995&i=2> | 
@kedarsdixit | M +91 90499 15588 | T +91 (20) 6703 4783

Persistent Systems | Partners In Innovation | 
www.persistent.com<http://www.persistent.com>

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

--

Regards,

Vaquar Khan

+1 -224-436-0783

Greater Chicago

--

Twitter: https://twitter.com/holdenkarau





If you reply to this email, your message will be added to the discussion below:

http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html

To start a new topic under Apache Spark User List, email 
ml+s1001560n1...@n3.nabble.com<mailto:ml+s1001560n1...@n3.nabble.com>
To unsubscribe from Apache Spark User List, click 
here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>.
NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
Twitter: https://twitter.com/holdenkarau

Re: Use of Accumulators

2017-11-14 Thread Holden Karau 
And where do you want to read the toggle back from? On the driver?

On Tue, Nov 14, 2017 at 3:52 AM Kedarnath Dixit <
kedarnath_di...@persistent.com> wrote:

> Hi,
>
>
>
> Inside the transformation if there is any change for the Variable’s
> associated data, I want to just toggle it saying there is some change while
> processing the data.
>
>
>
> Please let me know if we can runtime do this.
>
>
>
>
>
> Thanks!
>
> *~Kedar Dixit*
>
> Bigdata Analytics at Persistent Systems Ltd.
>
>
>
> *From:* Holden Karau [via Apache Spark User List] [
> mailto:ml+s1001560n29995...@n3.nabble.com
> ]
> *Sent:* Tuesday, November 14, 2017 1:16 PM
> *To:* Kedarnath Dixit 
>
>
> *Subject:* Re: Use of Accumulators
>
>
>
> So you want to set an accumulator to 1 after a transformation has fully
> completed? Or what exactly do you want to do?
>
>
>
> On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=29995&i=0>> wrote:
>
> Confirmed ,you can use Accumulators :)
>
>
>
> Regards,
>
> Vaquar khan
>
> On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=29995&i=1>> wrote:
>
> Hi,
>
>
>
> We need some way to toggle the flag of  a variable in transformation.
>
>
>
> We are thinking to make use of spark  Accumulators for this purpose.
>
>
>
> Can we use these as below:
>
>
>
> Variables  -> Initial Value
>
>  Variable1 -> 0
>
>  Variable2 -> 0
>
>
>
> In one of the transformations if we need to make Variable2's value to 1.
> Can we achieve this using Accumulators? Please confirm.
>
>
>
> Thanks!
>
>
>
> With Regards,
>
> *~Kedar Dixit*
>
>
>
> [hidden email] <http:///user/SendEmail.jtp?type=node&node=29995&i=2> |
> @kedarsdixit | M  value="+919049915588" target="_blank">+91 90499 15588 | T  value="+912067034783" target="_blank">+91 (20) 6703 4783
>
> *Persistent Systems | **Partners In Innovation** | www.persistent.com
> <http://www.persistent.com>*
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
> --
>
> Regards,
>
> Vaquar Khan
>
> +1 -224-436-0783
>
> Greater Chicago
>
> --
>
> Twitter: https://twitter.com/holdenkarau
>
>
> --
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html
>
> To start a new topic under Apache Spark User List, email
> ml+s1001560n1...@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-- 
Twitter: https://twitter.com/holdenkarau

RE: Use of Accumulators

2017-11-14 Thread Kedarnath Dixit 
Hi,

Inside the transformation if there is any change for the Variable’s associated 
data, I want to just toggle it saying there is some change while processing the 
data.

Please let me know if we can runtime do this.


Thanks!
~Kedar Dixit
Bigdata Analytics at Persistent Systems Ltd.

From: Holden Karau [via Apache Spark User List] 
[mailto:ml+s1001560n29995...@n3.nabble.com]
Sent: Tuesday, November 14, 2017 1:16 PM
To: Kedarnath Dixit 
mailto:kedarnath_di...@persistent.com>>
Subject: Re: Use of Accumulators

So you want to set an accumulator to 1 after a transformation has fully 
completed? Or what exactly do you want to do?

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan <[hidden 
email]> wrote:
Confirmed ,you can use Accumulators :)

Regards,
Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <[hidden 
email]> wrote:

Hi,



We need some way to toggle the flag of  a variable in transformation.



We are thinking to make use of spark  Accumulators for this purpose.



Can we use these as below:



Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0



In one of the transformations if we need to make Variable2's value to 1. Can we 
achieve this using Accumulators? Please confirm.



Thanks!



With Regards,

~Kedar Dixit


[hidden email] | @kedarsdixit | M 
+91 
90499 15588 | T +91 (20) 6703 4783

Persistent Systems | Partners In Innovation | 
www.persistent.com<http://www.persistent.com>
DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
--
Twitter: https://twitter.com/holdenkarau


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Use-of-Accumulators-tp29975p29995.html
To start a new topic under Apache Spark User List, email 
ml+s1001560n1...@n3.nabble.com<mailto:ml+s1001560n1...@n3.nabble.com>
To unsubscribe from Apache Spark User List, click 
here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=a2VkYXJuYXRoX2RpeGl0QHBlcnNpc3RlbnQuY29tfDF8Njk0OTY1MzYz>.
NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

Re: Use of Accumulators

2017-11-13 Thread Holden Karau 
So you want to set an accumulator to 1 after a transformation has fully
completed? Or what exactly do you want to do?

On Mon, Nov 13, 2017 at 9:47 PM vaquar khan  wrote:

> Confirmed ,you can use Accumulators :)
>
> Regards,
> Vaquar khan
>
> On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <
> kedarnath_di...@persistent.com> wrote:
>
>> Hi,
>>
>>
>> We need some way to toggle the flag of  a variable in transformation.
>>
>>
>> We are thinking to make use of spark  Accumulators for this purpose.
>>
>>
>> Can we use these as below:
>>
>>
>> Variables  -> Initial Value
>>
>>  Variable1 -> 0
>>
>>  Variable2 -> 0
>>
>>
>> In one of the transformations if we need to make Variable2's value to 1.
>> Can we achieve this using Accumulators? Please confirm.
>>
>>
>> Thanks!
>>
>>
>> With Regards,
>>
>> *~Kedar Dixit*
>>
>> kedarnath_di...@persistent.com | @kedarsdixit | M +91 90499 15588
>> <+91%2090499%2015588> | T +91 (20) 6703 4783 <+91%2020%206703%204783>
>>
>> *Persistent Systems | **Partners In Innovation** | www.persistent.com
>> *
>> DISCLAIMER
>> ==
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>
>
>
> --
> Regards,
> Vaquar Khan
> +1 -224-436-0783
> Greater Chicago
>
-- 
Twitter: https://twitter.com/holdenkarau

Re: Use of Accumulators

2017-11-13 Thread vaquar khan 
Confirmed ,you can use Accumulators :)

Regards,
Vaquar khan

On Mon, Nov 13, 2017 at 10:58 AM, Kedarnath Dixit <
kedarnath_di...@persistent.com> wrote:

> Hi,
>
>
> We need some way to toggle the flag of  a variable in transformation.
>
>
> We are thinking to make use of spark  Accumulators for this purpose.
>
>
> Can we use these as below:
>
>
> Variables  -> Initial Value
>
>  Variable1 -> 0
>
>  Variable2 -> 0
>
>
> In one of the transformations if we need to make Variable2's value to 1.
> Can we achieve this using Accumulators? Please confirm.
>
>
> Thanks!
>
>
> With Regards,
>
> *~Kedar Dixit*
>
> kedarnath_di...@persistent.com | @kedarsdixit | M +91 90499 15588
> <+91%2090499%2015588> | T +91 (20) 6703 4783 <+91%2020%206703%204783>
>
> *Persistent Systems | **Partners In Innovation** | www.persistent.com
> *
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>



-- 
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago

Use of Accumulators

2017-11-13 Thread Kedarnath Dixit 
Hi,


We need some way to toggle the flag of  a variable in transformation.


We are thinking to make use of spark  Accumulators for this purpose.


Can we use these as below:


Variables  -> Initial Value

 Variable1 -> 0

 Variable2 -> 0


In one of the transformations if we need to make Variable2's value to 1. Can we 
achieve this using Accumulators? Please confirm.


Thanks!


With Regards,

~Kedar Dixit


kedarnath_di...@persistent.com | @kedarsdixit | M +91 90499 15588 | T +91 (20) 
6703 4783

Persistent Systems | Partners In Innovation | www.persistent.com
DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.