[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303086#comment-17303086 ] Yashwanth Kumar commented on SPARK-10697: - Glad that the change proposed by me 5 yrs back got resolved. Sorry that apache account got disabled. Got a new one now. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Assignee: Marco Gaido >Priority: Minor > Fix For: 2.4.0 > > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592899#comment-16592899 ] Apache Spark commented on SPARK-10697: -- User 'mgaido91' has created a pull request for this issue: https://github.com/apache/spark/pull/22236 > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577886#comment-16577886 ] Akhil Sakarwal commented on SPARK-10697: Is there any update on this issue . By when can we expect to see lift in FP algorithm using mllib ? It will be usefull feature to measure relevance of association rules along with confidence. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447898#comment-16447898 ] Mourits de Beer commented on SPARK-10697: - Hello - is this still in progress? We require the lift of an association rule for a production use case. We can solve it for our own end, but it would be more elegant to build the lift method into the AssociationRules class, and it feels like that should be fairly trivial. Surprising (yet understandable) that it's been unresolved for 2.5 years. If a pull request is still necessary, I'll be happy to figure it out. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359639#comment-16359639 ] Sean Owen commented on SPARK-10697: --- Yes, I think it's OK to add. Go ahead and propose a PR. Lift is confidence, normalized for the prior probability of observing the antecedent at all. Yes it is the right tool when evaluating rules vs each other for interest. It's a likelihood ratio. Confidence is of interest when you know you have the antecedent (e.g. already added those items to a basket) and want to know about consequents. There the prior probability would be irrelevant. You can compute lift from confidence but it's extra work and so does make some sense to compute this along the way. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[
https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359623#comment-16359623
]
Tristan Stevens commented on SPARK-10697:
-
[~srowen] a big +1 from me to implementing this. Without Lift, it becomes very
difficult to assess whether a rule is even worth looking at.
As an example, using the dataset from Wikipedia, we get the following output
currently:
{{from pyspark.ml.fpm import FPGrowth}}{{df = spark.createDataFrame([}}
{{ (0, ["milk", "bread"]),}}
{{ (1, ["butter"]),}}
{{ (2, ["beer", "diapers"]),}}
{{ (3, ["milk", "bread", "butter"] ),}}
{{ (4, ["bread"],)}}
{{], ["id", "items"])}}{{fpGrowth = FPGrowth(itemsCol="items", minSupport=0.2,
minConfidence=0.2)}}
{{model = fpGrowth.fit(df)}}{{# Display frequent itemsets.}}
{{model.freqItemsets.show()}}{{# Display generated association rules.}}
|items|freq|
|[milk]|2|
|[milk, butter]|1|
|[milk, butter, br...|1|
|[milk, bread]|2|
|[diapers]|1|
|[diapers, beer]|1|
|[bread]|3|
|[butter]|2|
|[butter, bread]|1|
|[beer]|1|
{{model.associationRules.show()}}
|antecedent|consequent|confidence|
|[milk]|[butter]|0.5|
|[milk]|[bread]|1.0|
|[milk, butter]|[bread]|1.0|
|[beer]|[diapers]|1.0|
|[bread]|[milk]|0.|
|[bread]|[butter]|0.|
|[milk, bread]|[butter]|0.5|
|[diapers]|[beer]|1.0|
|[butter, bread]|[milk]|1.0|
|[butter]|[milk]|0.5|
|[butter]|[bread]|0.5|
However this misses the detail that milk->bread is much less interesting than
diapers->beer. When we add in lift we get the following:
|antecedent|consequent|confidence|lift|
|[milk]|[butter]|0.5|1.25|
|[milk]|[bread]|1.0|1.|
|[milk, butter]|[bread]|1.0|1.|
|[beer]|[diapers]|1.0|5.0|
|[bread]|[milk]|0.|1.|
|[bread]|[butter]|0.|0.8333|
|[milk, bread]|[butter]|0.5|1.25|
|[diapers]|[beer]|1.0|5.0|
|[butter, bread]|[milk]|1.0|2.5|
|[butter]|[milk]|0.5|1.25|
|[butter]|[bread]|0.5|0.8333|
So the proposal would be to add Lift to the Rules class, calculated by
{{lift( x => y ) = sup(x U y) / (sup( x ) * sup( y ))}}
> Lift Calculation in Association Rule mining
> ---
>
> Key: SPARK-10697
> URL: https://issues.apache.org/jira/browse/SPARK-10697
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
>Reporter: Yashwanth Kumar
>Priority: Minor
>
> Lift is to be calculated for Association rule mining in
> AssociationRules.scala under FPM.
> Lift is a measure of the performance of a Association rules.
> Adding lift will help to compare the model efficiency.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300072#comment-16300072 ] R Stegmaniac commented on SPARK-10697: -- Is this still in the pipeline? Would then also suggest to add leverage at same time. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610163#comment-15610163 ] Shreya Agarwal commented on SPARK-10697: [~yashkumar1992] Hey, so is someone fixing this? > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469830#comment-15469830 ] Daniel Müller commented on SPARK-10697: --- Yes, this should be a new attribute for the Rule class, as the lift factor is calculated separately for each rule. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469821#comment-15469821 ] Sean Owen commented on SPARK-10697: --- Would this be an addition to the Rule class? a little more detail on how you propose to add it would be helpful to evaluate whether this is something worth changing. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469791#comment-15469791 ] Daniel Müller commented on SPARK-10697: --- @[~yashkumar1992]Is it possible to open this ticket again? Or should I create a new one? I'm new to JIRA... Thank you > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469778#comment-15469778 ] Daniel Müller commented on SPARK-10697: --- It really would make sense to add the lift calculation to the Association rule mining, as the lift is one important characteristic number. lift (X --> Y) = support ( X U Y) / (support (X) x support (Y)) See also https://en.wikipedia.org/wiki/Association_rule_learning#Lift Could you please re-open the ticket as this feature would be really helpful? > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081005#comment-15081005 ] Yashwanth Kumar commented on SPARK-10697: - Dear Sean Owen, My idea is to add Lift (in addition to confidence) for the rules generated for association rules. I can provide additional details if needed. Thank you > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081003#comment-15081003 ] Yashwanth Kumar commented on SPARK-10697: - Dear Sean Owen, My idea is to add Lift (in addition to confidence) for the rules generated for association rules. I can provide additional details if needed. Thank you > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080996#comment-15080996 ] Yashwanth Kumar commented on SPARK-10697: - Adding Lift as an extra parameter to the Genrated Association rules. > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081002#comment-15081002 ] Yashwanth Kumar commented on SPARK-10697: - Dear Sean Owen, My idea is to add Lift (in addition to confidence) for the rules generated for association rules. I can provide additional details if needed. Thank you > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10697) Lift Calculation in Association Rule mining
[ https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081001#comment-15081001 ] Yashwanth Kumar commented on SPARK-10697: - Dear Sean Owen, My idea is to add Lift (in addition to confidence) for the rules generated for association rules. I can provide additional details if needed. Thank you > Lift Calculation in Association Rule mining > --- > > Key: SPARK-10697 > URL: https://issues.apache.org/jira/browse/SPARK-10697 > Project: Spark > Issue Type: New Feature > Components: MLlib >Reporter: Yashwanth Kumar >Priority: Minor > > Lift is to be calculated for Association rule mining in > AssociationRules.scala under FPM. > Lift is a measure of the performance of a Association rules. > Adding lift will help to compare the model efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
