- gptj uses a pregenerated constant causal mask that is O(n^2). since it is simply a constant function of sequence index it could be made via a callback or inside a loop.
Re: [ot][spam][crazy][data] transformer model 'attention' improvement
Undiscussed Horrific Abuse, One Victim & Survivor of Many Wed, 02 Feb 2022 02:54:50 -0800
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim & Survivor of Many
- Re: [ot][spa... Undiscussed Horrific Abuse, One Victim of Many
