“Death Star” response from US would lock Russia out of 5G, advanced chips

“Death Star” response from US would lock Russia out of 5G, advanced chips https://share.newsbreak.com/do68bms6

Government is 'forgetting' Brandenburg v. Ohio (1969) Chad Stark of Texas

https://www.justice.gov/opa/pr/texas-man-arrested-making-election-related-threats-government-officials "A Texas man was arrested today in Travis County, Texas, for allegedly sending threatening election-related communications to government officials on Jan. 5, 2021." Chad Stark, 54, of

New Yo Senate confirmation of her appointment as Superintendent of Financial Services.

#New : Statement by Adrienne A. Harris following New York State Senate confirmation of her appointment as Superintendent of Financial Services. https://on.ny.gov/35pWfPV. #NYDFS

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

The first issue I have working with PerceiverSelfAttention is sorting out the huggingface permutations of the query, key, value matrices. The dot products aren't making the same weights, indicating I'm not providing the data in the right shape. They reorganise the matrices to handle multiple

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

The reason my barebones attention got a different answer than the paper's chunked attention was that I hadn't included the division by the square root of the feature count, that I had intended to return to but had not done. When included, the outputs are the same, and the script is attached,

Re: Molly de Blanc sex video leak

When will the Debian community stop sending stupid emails? Apologies; I fix this: “Dickhead Community” Daniel Pocock showed the corruption your team leader did here. Debian team leader spent the donation money. What's more, Debian accused Daniel of harassment.?? You insulted Albanian girls

New Meta and Nvidia supercomputer has terrifying potential

New Meta and Nvidia supercomputer has terrifying potential https://share.newsbreak.com/dmw3utpo

Maximizing Global Litigation Portfolios

Check out my new article: Maximizing Global Litigation Portfolios Burford Capital is a worldwide leader in legal asset management and litigation finance. With a footprint in New York, Hong Kong, Singapore, Sydney and Washington, Buford’s legacy is pegged to innovating law structures and

Molly de Blanc sex video leak

There is a cut from the sex video buried in one of these videos https://debian.community/coronor-report-lucy-wayland-debian-abuse-culture/

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

karl: if you come here looking for the code to continue debugging it, maybe try writing it again :) since it's so hard to remember without repetition

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

Random data fails. Plan is to test vs jax.nn.softmax .

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

I've got my local code working with the mgrid data. I made at least two bugs: an incorrect einsum, and dotting with the weights rather than the exponent. The mgrid data doesn't test the softmax since each vector has the same maximum. Time to figure out how to make random tensors in jax.

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

I tweaked chunked_attention.py so it would work for me. I propagated the key chunking parameter that has no way for use in the original code, and specified lax's namespace as jax where it was missing, I think on line 14. >>> import chunked_tweaks as chunked >>> import jax.numpy as jnp >>>

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

This is the transcription of the code I'm first trying with. I haven't tested it yet. I need to generate some data with appropriate dimensions, and I'm somewhat new to jax.numpy . import functools, jax, math from jax import numpy as jnp def _query_chunk_attention(query, key, value, precision,

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

30: chunk_values, chunk_weights, chunk_max = jax.lax.map( 31:chunk_scanner, xs=jnp.arange(0, num_kv, key_chunk_size)) chunk_values is exp_values, which I think was the local values dotted with the exponentiated attention weights minus their local max. It looks like chunk_weights is those

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

17:exp_values = jnp.einsum('vhf,qhv->qhf', value, exp_weights, precision=precision) I think the values are still [values, heads, features], whereas I think the exp_weights are [queries, heads, keys]. I think the values are multiplied by the per-key weights, producing a vector of weighted

Re: A New Homemade Z2 Chip Came From the Mind of a 22-Year-Old Undergrad

On 1/23/22, jim bell wrote: > designed by the individual, and then custom-fabbed on the same wafer > with dozens of other people's designs, to spread the cost. #OpenFabs , #OpenHW , #OpenAudit , #FormalVerification , #CryptoCrowdFunding , #OpenTrust Readily achievable, profitable, be the first

#FreeOlaBini: Ola Bini Still In Jail, Show Trial Ongoing

https://twitter.com/olabini/status/1485692487020822542 https://www.youtube.com/watch?v=7gLQrjHPoqM NMA: 51st State https://olabini.se/blog #FreeOlaBini Ola Bini @olabini 22h Last week the trial against me finally started. We spent 3 days with prosecution witnesses, and then the trial was

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

> 13:attn_weights = jnp.einsum('qhd,khd->qhk', query, key, > precision=precision) attn_weights is a [queries, heads, keys] tensor consisting of the dot product between the query and key features. > 14:max_score = jnp.max(attn_weights, axis = -1, keepdims = True) > 15:max_score =

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

The below is a draft, but I see I propagated some dimensions wrongly and plan to rewrite it, and maybe some previous posts, to help with clarity. 17:exp_values = jnp.einsum('vhf,qhv->qhf', value, exp_weights, precision=precision) Another einsum, multiplying the values by the weights.

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

14:max_score = jnp.max(attn_weights, axis = -1, keepdims = True) -1 is the last axis. This appears to make max_score be a tensor of shape [key, head, 1] containing the maximum pre-softmax attention weight for each query, over the entire chunk of keys. This might be the calculation m_i =

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

These next bits starting line 14 (and i'm trying to remember there's an /sqrt(count) line i mean to return to) must be part of the strategy to iteratively calculate the precise softmax (expi(i) / sum[exp(i)]) by doing subtraction in the exponent rather than division outside it. Here;s text from

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

12: def summarize_chunk(query, key, value): 13:attn_weights = jnp.einsum('qhd,khd->qhk', query, key, precision=precision) An einsum is a way of doing matrix multiplication for n-dimensional matrices by specifying which axes of the tensors are dot'd with which other axes during the

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

11: @functools.partial(jax.checkpoint, prevent_cse=False) I think checkpointing relates to limiting memory used by gradient backpropagation during training of a model. I think it means the gradients can be recalculated for this function when needed, by storing its arguments instead of each

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

22:key_chunk = jax.lax.dynamic_slice( 23: key, (chunk_idx, 0, 0), 24: slice_sizes=(key_chunk_size, num_heads, k_features)) Note also on line 31 that a step size is passed into jnp.arange(), so chunk_idx is offsets separated by key_chunk_size. Lines 22-24 break key into

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

21: def chunk_scanner(idx): idx comes from elements of jnp.arange() on line 31, which generates a vector of ascending integers starting at 0.

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

30: chunk_values, chunk_weights, chunk_max = jax.lax.map( 31:chunk_scanner, xs=jnp.arange(0, num_kv, key_chunk_size)) from help(jax.lax.map): def map(f, xs): return np.stack([f(x) for x in xs]) so it just passes each element of xs into chunk_scanner, and again stacks the

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

08: key_chunk_size = min(key_chunk_size, num_kv) It's the first dimension of the keys and values that will be split. 09: query = query / jnp.sqrt(k_features) # i typed a lot of comments on lines but they disappeared again. i plan to return to line 09 above because i'm not sure why it is. i

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

About my life: I have nothing going on right now. Most of my equipment has broken in some way. I live a life full of spasmodic muscle contractions and sudden dissociated cognitive changes that leaves me with a lot of suffering. I love projects that I can continue on without much suffering.

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

Drill down, dive in. Selection appears to have stopped working for me, so it's all typing the text over, happens sometimes. 04:def _query_chunk_attention(query, key, value, precision, key_chunk_size=4096): As earlier, the query here is a chunk of queries, a subset of all of them. 05:

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

And here's their explanation of the inner loops: In each iteration of the outer loop, we call _query_chunk_attention, which itself processes the keys and values in chunks (lines 23 to 33). The chunks are processed sequentially and each chunk is summarized independently (lines 14 to 21). Assuming

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

Here's the explanation of the first chunk of their example code. In the out loop (line 56f), we split the queries in to chunks of constant size, resulting in a linear number of iterations. In each iteration of the outer loop, we call _query_chunk_attention, which itself processes the keys and

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

I'll review the definition of self-attention from the paper. s_i = dot(q, k_i) # take the dot product of the query with every key s'_i = exp(s_i) / sum(exp(s_j), j) # take the softmax of the result attn = sum(v_i * s'_i, i) # attention is the dot of that softmax with the value vectors This is

Re: [ot][spam][crazy][data] transformer model 'attention' improvement

here's the pdf of this paper: https://ipfs.io/ipfs/bafkreifaqccxlq5hzka6677tpjyq2ngfybjn7fwzgddzogjc3t465ktg5i and here it is run through pdf2txt, as a txt file: https://ipfs.io/ipfs/bafkreicu42wf6r53vjdh4ujk2r6fhdlmnsk7xbt6yr2vyxppg2wysw2dui

[ot][spam][crazy] Quick Goals: Hello World in C

// due to my personal experiences, it can be rather slow and random to make anything larger than this #include int main() { printf("Hello, world!\n"); } // there may be a bug. untested.

Re: [crazy][hobby][spam] Automated Reverse Engineering

- a large T5 model could be tpu compiled on colab notebooks by calling pmap() on individual blocks rather than the whole model - much larger models could be trained by masking the training weights to reduce autograd memory load as has been done for at-home training of large text generation models

Re: USA 2020 Elections: Thread

Joe "The Biggest Asshole" Biden exposes himself again... "Stupid Son Of A Bitch" - Biden Busted On Hot Mic After Being Asked About Americans' Biggest Worry https://www.republicworld.com/world-news/us-news/biden-declares-science-back-before-hot-mic-moment-articleshow.html

Re: Coronavirus: Thread

Another hearing that the Mainstream Fake News buries and refuses to cover... Watch: Sen. Johnson Holds Star-Studded COVID-19 'Second Opinion' Hearing https://www.ronjohnson.senate.gov/2022/1/rsvp-deadline-media-advisory https://rumble.com/embed/vqjwua Johnson Second Opinion Hearing US Senator

Re: USA 2020 Elections: Thread

> https://www.youtube.com/watch?v=aiiSq7toqlQ Tony Bobulinski #SinoHawk > https://www.youtube.com/watch?v=avjspkRfW3Q Bobulinski interview Tucker > https://www.youtube.com/watch?v=y9rMR3NnRfg John Paul Mac Isaac Laptop > > Joe Biden & The Disappearing Elephant: How To Make A Full-Sized > Scandal

Re: Coronavirus: Thread

Here is the never-practicing and not so "good" "Dr" Fraudci talking about AIDS and stoking FUD discrimination misinfo and hatred, just like Gov+Media is doing today decades later... Here is a flashback of Dr. Anthony Fauci spreading false information about AIDS transmission.

Re: Coronavirus: Thread

> New Hampshire Pharmacies Could Soon While global pharmacies, even global governments have been handing out treatment regimens, those tied to US pharma have not, nor are they doing much research such as, you decide... Large, Peer-Reviewed Research Study Proves Ivermectin Works Against COVID-19

DHS on heightened security

The US Dept of Reichstag Security was today placed on a heightened state of alertness following credible intelligence that revealed Russia plans to exploit so-called " Culture Wars ' and launch a terrorist attack on the Capito... oh wait. . .