I think you need a much larger training set. Are you really writing this in python?
On Sun, Jul 11, 2021, 7:43 PM <[email protected]> wrote: > https://pypi.org/project/aitextgen/ > To do it yourself: Go to the link on that page above called: Train a > GPT-2 model + tokenizer from scratch (GPU) > <https://colab.research.google.com/drive/144MdX5aLqrQ3-YW-po81CQMrD6kpgpYh?usp=sharing> > > 100KBs of enwik8.txt, here is some tests. Training took like 10 mins using > a GPU on colab. > > Parameters are: > config = build_gpt2_config(vocab_size=5000, max_length=32, dropout=0.0 > , n_embd=256, n_layer=8, n_head=8) > config > > and, batch_size=256, > > Bold text is my prompt, it tried it multiple times. It seems to mostly > copy directly from the dataset, at least if trained on just 100KBs. > > > > *My dog loves to* as having "low-functioning autism" (LFA), whi > ========== *My dog loves to* assert it. I make my war upon privilege and > authority, ========== *My dog loves to* the point of the state as a means > of regulating violence, so that the ========== *My dog loves to* the page > to keep this article free of clutter --> ========== *My dog loves to* > the page to keep this article free of clutter --> ========== *My dog > loves to* as having "low-functioning autism" (LFA), whi > ========== *My dog loves to* the present day'', 1945.</ref> > [[Diggers ========== *My dog loves to* the ''real increase'' would focus > more attention (and research fun ========== *My dog loves to* as having > "low-functioning autism" (LFA), whi ========== *My dog loves to* > the page to keep this article free of clutter --> > > *He was driving* in [[Randolph Bourne]]'s view that "war is the heal > ========== *He was driving* in response to Bolshevik policy and the > suppression of the Kr ========== *He was driving* in response to > Bolshevik policy and the suppression of the Kr ========== *He was driving* > in [[Barcelona]], in [[Catalonia]]. Militant [[antif ========== *He was > driving* in response to Bolshevik policy and the suppression of the Kr > ========== *He was driving* in response to Bolshevik policy and the > suppression of the Kr ========== *He was driving* in [[Barcelona]], in > [[Catalonia]]. Militant [[antif ========== *He was driving* in response > to Bolshevik policy and the suppression of the Kr ========== *He was > driving* in response to Bolshevik policy and the suppression of the Kr > ========== *He was driving* in response to Bolshevik policy and the > suppression of the Kr > > *This is not so* in themselves signs of autism or Asperger's syndrome. > Others assert that it is ========== *This is not so* in themselves signs > of autism or Asperger's syndrome. Others assert that it is ========== *This > is not so* still thinking in a community]] [[M-I, [[I]]'s synthes > ========== *This is not so* in themselves still think of autism as a > communication disorder so severe that no sp ========== *This is not so* > in 'asssly anarchy; I am an individualist. She wishes to destroy the right > ========== *This is not so*ftware movement]]. These cyber-communities > include the [[GNU]], [[L ========== *This is not so* intense from > childhood that many of them have [[imaginary friend]]s as companion > ========== *This is not so* in related [[social movement]]s) that > advocate the elimination of authoritarian institution ========== *This is > not so* in society. However, the term to creation of Anarchist > Communists]] in the northeastern ========== *This is not so* still thinks > that autism is pursued (see references below), including Dr > > > > GPT-2 had: > Vocabulary of 50,257 tokens was used. > Batch size of 512. > Context window of 1024 tokens were used. > 48 layers. > 1600 dimensional vectors for word embedding. > --------- > I would say probably the number of layers could be higher for my test? 16 > conked out on my colab though. > > > > Now I try 6MB, The Brown Corpus, a higher quality dataset too. Parameters > are now: config = build_gpt2_config(vocab_size=5000, max_length=128 > , dropout=0.0, n_embd=256, n_layer=8, n_head=8) > config > > and, batch_size=256, > > It looks much better. Woah. Clearly visible now. Much less matches in the > dataset, like very few 1 or 2 words at most. Still funky like mine a bit > interesting I will check later. > > It took 1.2 hours using a GPU lol. Mine would take 1 minute if in C++ > using CPU 1 core. > > I'll try 1MB tomorrow, these parameters and from TBC (same dataset). > > > > *We were walking* on a day". Saude louder, Mama said. Elaine rejecting > the smooth surprised chiefly against the bar side, which was looking at the > bottom of a side, the two input according to the radish, and what William > was mailing looking left on the corn orchestra with a "comoon free > ========== *We were walking* in around it, quietly. Nonetheless, the > Ozagua with the rifle, want of the great spangle and joy of as venerant > about it. I gave him a chance to unload the flow of doork. When Linda Kay > took him into he was there Linda Kay toward me awake in the hall, he was on > what madness- ========== *We were walking* up, and he took one love the > breath. #HEADA'OR RESEARD FOR RES:# Neither paused by the Geology of > Medical Maine and American both of anything novel in Wisharge from a jury > to matron, and he halfbarly began to three stryses in the house ========== *We > were walking* in a single flight in August **h. Here the picket > underscopilco and members of the St& Peter's Iows, will not wear out all > upset and worn breakdowns throbbering tennissue passing. This may perhaps > costs of insulation not to protect him to invite females, to prob > ========== *We were walking* to Johnston getting an editor of the ball > player's hands wreckoning each spring. @ #TEW RULS OVERT# _TO THE EDITOR:_ > "Doy you know", the liberal editor of a Hriselber Vice President > Eisenhower. And for more ========== *We were walking* up in two weeks > ears". President Kennedy live in the day of the country chair repere was in > keeping a minute, felt there. They all practically children learned become > already mentally. He states which Mr& Joe drove of his color was a > permitted him to, and he felt unqu ========== *We were walking* in a way > and rounded you wouldn't have had handed between this harness and Mantle is > so far for sinners eating. Its enjoyment of that crystallic meditatisfy had > himed Dowago. As Calcutta mouse on Wary outdoorsas remain until Mando could > see the virer's mood mind at his red ========== *We were walking* in that > kind of life. Everything proud and his attempted to use. If he doubted God > held the side of the side, he then walked up the street. I waited into the > pathetic waters of the Eve, piano Brofty Rector, firmlyly and a host. <Oh, > wanted him back in the batting hoo ========== *We were walking* in him. > When he was four or words to say, all he likes you in? "You're in the > corniest to corrupt", the soa was bullets turned out. "You just look like > yours", he said, watching young child again to pull a place for a room. > Means and George ========== *We were walking* to Soviet Paula. "Anyone > smart a way to sink in time. You certainly want to back over to the Brooks > Adams and Rover, the modern street outside the Dread. In any case, Player > might well have hesday-chool-saffy story which is ended. Still, the company > f > > *He was driving* mocking, and obviously that it was dirty. We were not > scarning as if we were watered. The water leveled and weathered the sense > of pronounced himself. All too is salary for guard against the rustland > flying grave matches to the lost, and he lived for Raphael with a reform. > We ========== *He was driving* him the fuzzy into rooms and waiting on > his back toward the nearest boots, and the mountains to say they had six > lodes. Or in these phrases in their own country and qualities. Every night > that we had been at first crowd and broken slack horses. There were other > slides no different ========== *He was driving* him around the street and > the gray razing repeated square. They were ridely through tracking night, > they would never have fled, stern striped him to stop, the letter into the > house. They looked at. They cracked, and a puzzled loczed, their rough cuts > and folded dollars, and adapt ========== *He was driving* him against the > church. The Alfred Eve were shot saying. The same cancer liked him, he was > sentry blurned to the right alone. He saw carefully as he fell as a boat > feet, and it turned out. That was it. This. He smiled. When I said, "I > could hardly we'd get ========== *He was driving* out the water tongue > and the water he came in a sinner, but he does read it back and for fear of > a bat. Alex's door scanned the sensual and sag, a little more carriage, but > as my made me a strangely darker. When he chose a very early in that joyous > measure, irony I'll be open ========== *He was driving* back into > bathing, brushedbugs, and the dust flags, branches beckons. Later in the > switch the three multiple left, eyes and waiting feathing repeats which > before greeted him. An intense disturra; a man of was gin story, who was > pungary, probably in his fo ========== *He was driving* out from his chin > and himself. He noticed his mouth, unicesaid, but he would still in the > great deeper racing dry. As he lodge, the blonde where men and neither go > and I'd fell when Meltzer as they had turned it out of ball, and declared > it would. If Bob McFotley was ========== *He was driving* out into the > cold and pawns from the protection of incorrect scatter residents now. > Piaou becoming a novel in shaft lays toer, the saddle boiling down and > feeling with louse in Maineaps, or the fox-lollator, with rough dogs, > waiting, or hair-sabb, or guarded over outs ========== *He was driving* > out from the road, snartridge the horns above the great straight ahead. The > only other spirit which had been doubled. That was the happiness the > historian driven concerned with the top non. When Foster stopped below the > church, hides by an unercected features that came nearever expected being > un ========== *He was driving* to him and great had sign him away in his > steps toward her. Once and Rose, he went more than honors and had done it. > He was delivering flat. He goulouded to think of brilliant; he had lunch > cover to wide him. Then he took off his two other spot flies, running it > down. > > *My dog loves to* ourselves as made as much. The swords, tears may be > able to get away from it; and the host profitating population down to be a > blood high level, in order to prevent it from being in relation to a lower > to extent problems. When the involved in the arts in rising suburbabine c > ========== *My dog loves to* penny in mortar and feeling. They looked > good returning, and good flesh. But they worried about that. No one had for > him about them tin the grand room, and I had never seen Rotte lion, and we > had dabard something about all the tract of one of the Callas bosses; for > three things that both sex ========== *My dog loves to*day. As a result, > the partnership responsibility for recruit and the federal courts for > interests and then through other ways to expose before dates are enemy > situated in the world. This is the establish within a common era and color > vent to create. You can find it in the third y ========== *My dog loves > to* attack dance in the house when the ends of Revenues remain prime with > contemporary **h". "Ah, we've got to go Henrietta so War", they had said a > tongue. He had caured into the lead. About half of the mounted Rector, > Massachusetts, M ========== *My dog loves to* the great men of the > twenty-year programs are flowed with liberated signs in Rhode Island. To > measures by protection from educational tend to edge already completes, and > moreover, more than likely to be thrust of needed. The enemy of this trend > abstraction is much less than I believe has ========== *My dog loves to* > crack "Remember Young Review", that pretty soon laid a smile background > profession. There was no evidence that Lao suvidenceia for tence attempts > to say her. "As it began, I'll send them them hold yourself", Sadiorm > editors. _AUST_ A " ========== *My dog loves to* college, in a certain > professionship with merit performance! When you are not abandoning a thing > for such, every representation toward me, represent the extreme shorts. > Your put itself, did not virtue the answer to its beginning not likely to > give up a picture of adjustment of middle ========== *My dog loves to* > capture the pressure and the summer of all postmasters, currently data from > a need for trio who had been no reasonable. Thus, there has been some > physical attitude: the same thought which has been little disappointing > this early last night, and it is to a way just when the young wooden yet > sheer became a com ========== *My dog loves to* discover, her parents had > brooded the big face. Most of the fingers and even delegates and reading > camps are had to do. ## Tom Hao considered himself had in the > wablenetration of Negro to the fiction. It had evident him associate now, > but he did not arrest the hou ========== *My dog loves to* be a suitable > place for a small purity, she learned that one phase away at which she was > told Lucy in due forty years. The reporters in turn- and still a touched > her- if only to recover the good, her didn't want her pay uneigh. "Can IS, > like that, so"- > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tb76967774bc4d8c2-M269abe9640add3d0772d4ca8> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tb76967774bc4d8c2-Md94a63694b0d00cbe6ab9e2a Delivery options: https://agi.topicbox.com/groups/agi/subscription
