Architectural Shader: Refraction Samples

[Byron Nash]

Why do more Samples in the Refraction section cause the object to get
darker? I would expect it to reduce the noisiness of the Gloss. I just need
to clean up the gloss of the refraction, am I looking in the wrong place?