[GitHub] sethah commented on issue #9777: [MX-9588] Add micro averaging strategy for F1 metric

2018-02-15 Thread GitBox 
sethah commented on issue #9777: [MX-9588] Add micro averaging strategy for F1 
metric
URL: https://github.com/apache/incubator-mxnet/pull/9777#issuecomment-366027233
 
 
   Thanks all for the review!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sethah commented on issue #9777: [MX-9588] Add micro averaging strategy for F1 metric

2018-02-14 Thread GitBox 
sethah commented on issue #9777: [MX-9588] Add micro averaging strategy for F1 
metric
URL: https://github.com/apache/incubator-mxnet/pull/9777#issuecomment-365664996
 
 
   Obviously, no change is expected since none of the update logic was changed 
here.
   
   **Before**
   ```
   Metric Data-Ctx  Label-Ctx   Data Size   Batch Size Output Dim   
  Elapsed Time
   
--
   F1 cpu(0)cpu(0)  131072  16 2
  0.82034
   
--
   F1 cpu(0)cpu(0)  131072  64 2
  0.46931
   
--
   F1 cpu(0)cpu(0)  131072  2562
  0.35179
   
--
   F1 cpu(0)cpu(0)  131072  1024   2
  0.35274
   
--
   F1 cpu(0)cpu(0)  16384   16 2
  0.10991
   
--
   F1 cpu(0)cpu(0)  16384   64 2
  0.057258
   
--
   F1 cpu(0)cpu(0)  16384   2562
  0.046497
   
--
   F1 cpu(0)cpu(0)  16384   1024   2
  0.044378
   
--
   F1 cpu(0)cpu(0)  204816 2
  0.01292
   
--
   F1 cpu(0)cpu(0)  204864 2
  0.0070581
   
--
   F1 cpu(0)cpu(0)  20482562
  0.005384
   
--
   F1 cpu(0)cpu(0)  20481024   2
  0.00511
   
--
   ```
   
   **After**
   ```
   Metric Data-Ctx  Label-Ctx   Data Size   Batch Size Output Dim   
  Elapsed Time
   
--
   F1 cpu(0)cpu(0)  131072  16 2
  0.81942
   
--
   F1 cpu(0)cpu(0)  131072  64 2
  0.46574
   
--
   F1 cpu(0)cpu(0)  131072  2562
  0.37653
   
--
   F1 cpu(0)cpu(0)  131072  1024   2
  0.32401
   
--
   F1 cpu(0)cpu(0)  16384   16 2
  0.099127
   
--
   F1 cpu(0)cpu(0)  16384   64 2
  0.057564
   
--
   F1 cpu(0)cpu(0)  16384   2562
  0.044527
   
--
   F1 cpu(0)cpu(0)  16384   1024   2
  0.04604
   
--
   F1 cpu(0)cpu(0)  204816 2
  0.012832
   
--
   F1 cpu(0)cpu(0)  204864 2
  0.0071719
   
--
   F1 cpu(0)cpu(0)  20482562
  0.0058281
   
--
   F1

[GitHub] sethah commented on issue #9777: Mx 9588

2018-02-12 Thread GitBox 
sethah commented on issue #9777: Mx 9588
URL: https://github.com/apache/incubator-mxnet/pull/9777#issuecomment-365147592
 
 
   Regarding other approaches, something I looked at was the following:
   
   ```python
   class MacroMetric(EvalMetric):
   
   def __init__(self, base_metric):
   super(MacroMetric, self).__init__("macro_" + base_metric.name, 
output_names=base_metric.output_names,
 
label_names=base_metric.label_names)
   self.base_metric = base_metric
   
   def update(self, labels, preds):
   self.base_metric.update(labels, preds)
   self.sum_metric += self.base_metric.get()[1]
   self.num_inst += 1
   self.base_metric.reset()
   ```
   
   Any metric that has defined the "micro" behavior can then be used as "macro" 
just by calling `metric = mx.metric.MacroMetric(mx.metric.F1())`, but that 
seems pretty awkward for the users, and also changes the default behavior to 
micro, which doesn't work for backwards compatibility. The current solution 
works well enough, and we could probably revisit in a later PR if needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services