That's over generalizing my point... Sendmail sucks at it, but I agree it does send files around. I would submit that's more a failure of the clients to do something smarter, or to make it easier not to send huge files inline (I hate when people do that). But the HUGE difference in this case is that sendmail has evolved to be highly distributed. I question whether Jabber really plays out that way, and whether us forcing it that way now would actually hamper its chances at success. As a service provider, I control a lot of users, and my choice of IM platform, in theory, is worth more than somecompany.com. As such, there is no way I'm going to choose a platform or implement a feature that causes me to double pay (or infinitely more if you weren't paying for client-server in the first place) for all bandwidth for files exchanged among my users. That free lunch ended with the millenium. So then does that give MSN and AOL an unnatural advantage over us? Not sure, but if I put myself in other service providers' shoes, it sure does.
Furthermore, most all meaningful email providers cap the message size, and at a number lower than the files we want to enable here... There's good reason for that, obviously. And even with that, the reality is most of us are falling over under the load of spam and mail bombs. In case peoples' inboxes don't show it, the spammers are winning right now, Sendmail can't handle them. Part of the reason they're winning is mistakes at the protocol level from 25 years ago... Unauthenticated senders, paths, headers, etc etc. So in this case we either pick a message size cap, and then have to implement something to allow files over that size (uh, how about client to client) or we let Jabber do what it should, which is route and message, and let someone else lug the heavy boxes. Add to that, as another person mentioned, that files are not the only instances of this issue. Are you proposing that we put H.323 video conferencing data in Jabber packets? The semantics of usage and delivery simply are not compatible between the two. XML is not and never will be about size or speed... I really think this is a non-starter, and if people disagree, we should probably take this conversation somewhere else, because this list has probably had enough of it. -----Original Message----- From: Michael F Lin [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 06, 2002 2:49 PM To: [EMAIL PROTECTED] Subject: RE: [JDEV] File transfers Firstly, there is no inherent problem with sending moderately large files through a software server. Sendmail does it all day, every day, on a massive scale, without relying on client-to-client connections. The trouble when we extend this to Jabber is that we expect Jabber to go faster than Sendmail, so it's not acceptable to block all other traffic while we send our file. Is this really a problem, though? I'm proposing - effectively, not literally - just open up a second Jabber session on both ends and do it that way. If the server is under heavy load, it can throttle the karma as necessary. If the recipient's queue is too large, it can reject the message. This is how e-mail works; just a bit smarter. This is a solution based on something that has been in place for years and is known to work reasonably well. E-mail has seen all the problems of mail bombing and spamming and denial of service attacks others have been warning of, but they have generally managed to keep themselves under control. I see no compelling reason to believe that the same will not be true for Jabber. Now, map what you and others are proposing we do back to e-mail. Every time you send an e-mail with an attachment, you first send a message to the recipient asking them for their IP address. Then your e-mail software opens up a connection with some custom protocol and transfers the file directly to the recpient. Firewalls are a problem, but you can set up some clever daemons and such to get around them most of the time. What if the recipient is not online at the time? What if it's a cell phone? What if, what if, what if...? Is that how e-mail should be done? If not, what's really so different about Jabber? Now, if you are transferring a gigabyte, you wouldn't send it via e-mail, so why would you send it via Jabber? If e-mail clients don't have facilities for client-to-client transfer of _extremely_ large files, why should Jabber clients? If you need to transfer huge files, use tools intended for that purpose. Maybe smart tools on the Jabber network could even help you figure out the right IP address for the recipient. But there is no reason to introduce this complexity to clients in order to transfer Word documents and Excel spreadsheets. -Mike |---------+----------------------------> | | Max Metral | | | <Max.Metral@peopl| | | epchq.com> | | | Sent by: | | | jdev-admin@jabber| | | .org | | | | | | | | | 06/06/2002 12:14 | | | PM | | | Please respond to| | | jdev | | | | |---------+----------------------------> >--------------------------------------------------------------------------- ---------------------------------------------------| | | | To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]> | | cc: | | Subject: RE: [JDEV] File transfers | | | | | >--------------------------------------------------------------------------- ---------------------------------------------------| There is no reasonable argument that client-to-client "open" file transfer via the Jabber servers is a good idea. You have to accept this to talk about what specific VERSIONS of that idea might be useful. The sub-10k file method isn't terrible, but it still has lots of abuse potential,and complicates client design with having to support multiple mechanisms. And I'm not sure what problem it's really solving other than the firewall situation. We need to solve the firewall problem for big files, there's nothing to say that solution can't handle small files too. So all in all, given that it's clear that client-server-client file transfer for normal size files is a dead end, I would say the merits don't warrant new techniques to get the server involved. -----Original Message----- From: Andy Beetz [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 06, 2002 8:00 AM To: '[EMAIL PROTECTED]' Subject: RE: [JDEV] File transfers I hear what you're saying, but there are ways you could easily stop large transfers (or any for that matter) from occuring (limit by filesize, limit bandwidth p/day or p/month, not allowing it at all full stop, content filtering etc etc). I'm not hearing anything constructive from you. Your use of the oxymoron "unofficial standard" (your other post) surprises me. For a technology that wants to be a standard, I would have expected it to try and accommodate usage of itself. The jabber server has it's ports registered with IANA why not register one for the clients? Then everyone is operating on the same level and not having to re-invent the wheel every time someone writes a client. All I'm trying to do is get something done which could prevent possible problems later on. -----Original Message----- From: Richard Dobson [mailto:[EMAIL PROTECTED]] Sent: 06 June 2002 12:11 To: [EMAIL PROTECTED] Subject: Re: [JDEV] File transfers Small level of abuse ???? The problems I outlines are not small they are very significant, there may be better ways to get copyrighted files but it does not stop people using jabber for that purpose now does it, and any copyrighted files that get transfered via the jabber server open the operators of that server to serious legal problems because they helped transfer the actual file between the two users, remember napster got shut down because they were helping users transfer files between each other, they werent even transfering the files in-band via the napster servers, which jabber already supports opening it to possible legal problems already, but transfering files via the servers just takes it up to a whole new level. Also transfering large files whether they are split up or not is just not feasable at all because of the major jump in bandwidth use for the server operators. Also there is nothing stopping someone from pushing a file on someone wether they wanted it or not, the only way files should be sent is that the sender sends a request to the receiver and then the receiver downloads the file from the sender (HTTP or otherwise), not the sender pushing the file to the receiver. Also if the whole reason for this is because of wanting to get around a firewall then you need to do it in the correct manor, using something like PASS, so if a server provider is prepared to allow the transfering of files this way they just setup a PASS server, you cant just force it on server providers like this, that would end up making lots of the free servers either shut down or start charging for using it. Bandwidth is not free. Richard ----- Original Message ----- From: "Andy Beetz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, June 06, 2002 10:03 AM Subject: RE: [JDEV] File transfers > I can see a small level of abuse perhaps, but there are better ways to > distribute/get hold of copyrighted files (kazaa to name but one). The > fact that the communications are 1 to 1 would not make the sharing of > files on a > massive scale feasible. I can only speak for myself obviously, but > I've only > ever used file transfer in msn messenger for small files. If I wanted > to download say a movie, I would use something designed for that > purpose, even > if it was from someone I knew I would find a way around not using an > IM app. > > I know in band data transfers present a problem, but I think splitting > the data would make it more server friendly. Plus the clients can > still send and > receive messages etc in between parts (given higher priority). > Firewalls present a major problem, but if you can get a connection to > the server, then > the problem dissipates. > > -----Original Message----- > From: Richard Dobson [mailto:[EMAIL PROTECTED]] > Sent: 06 June 2002 09:35 > To: [EMAIL PROTECTED] > Subject: Re: [JDEV] File transfers > > > I think that allowing file transfers of very small files in-band would > be cool, but anything over 10k or so should be sent out of band by > some other means, allowing it in band at all is also a big problem > because of the massive potential for abuse, in ways like DOS attacks > against individual clients and the server itself, excessive use of > expensive bandwidth, also creates copyright issues if people transfer > copyrighted files via the server > because it then brings the server providors into the line of fire > because they facilitated the transfer, etc etc. > > Because of all of these problems I dont think its a good idea to > transfer files in-band at all. > > Just my 2p > Richard > > ----- Original Message ----- > From: "Andy Beetz" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Thursday, June 06, 2002 6:58 AM > Subject: RE: [JDEV] File transfers > > > > What about the nntp idea for very large posts? Where the file is > > split > into > > several parts, each part being only small in size could be > > transmitted in-band just one at a time. As long as they carry header > > information the client at the other end should be able to decode and > > re-assemble. It > should > > be possible to request parts if they're missing. > > > > > > -----Original Message----- > > From: Michael F Lin [mailto:[EMAIL PROTECTED]] > > Sent: 05 June 2002 19:23 > > To: [EMAIL PROTECTED] > > Subject: Re: [JDEV] File transfers > > > > > > > > When we generalize the Jabber network to thousands of servers, it > > becomes something of a nightmare to transport stuff out of band. > > This is of course why HTTP is not too good for this purpose - too > > many people are behind firewalls. Any direct client-to-client > > connection with whatever protocol will of course have the same > > problem. Relying on e-mail routing is one option, but how do you > > negotiate what address to send an e-mail to? How do you receive it? > > What if you need a file but don't have access to your e-mail? > > > > There are any number of solutions you can set up with WebDAV and so > > forth, but what we would really, really like - particularly when it > > comes to > Jabber > > as a transport for web services - is a way to transport large > > payloads if not directly in-band, then in a band that fully adopts > > JID routing. > Jeremie > > has proposed PASS, which is a step forwards but not totally > > satisfactory. > > > > The only "good solutions" I've been able to think of basically > > involve running a Jabber server that knows how to route s2s on every > > client > machine. > > Which is, not coincidentally, something I'm working towards. > > > > -Mike > > > > > > > > |---------+----------------------------> > > | | Mike Oliver | > > | | <ollie@appsaspeer| > > | | s.com> | > > | | Sent by: | > > | | jdev-admin@jabber| > > | | .org | > > | | | > > | | | > > | | 06/05/2002 12:21 | > > | | PM | > > | | Please respond to| > > | | jdev | > > | | | > > |---------+----------------------------> > > > > > >--------------------------------------------------------------------- > >-- > >---- > > ---------------------------------------------------| > > | > > | > > | To: [EMAIL PROTECTED] > > | > > | cc: > > | > > | Subject: Re: [JDEV] File transfers > > | > > | > > | > > | > > | > > > > > >--------------------------------------------------------------------- > >-- > >---- > > ---------------------------------------------------| > > > > > > > > Why have just one protocol? > > > > SMTP does pretty well at file transfers that are asynch. The Jabber > > protocol can include a header for the attachments and the user at > > the > other > > > > end can decide if they want to download the file. The a request can > > then > be > > sent to the originating peer and an SMTP transfer begun and the > > remote client can notify the user when the transaction is complete > > by asking > where > > > > to put the file. There are SMTP libraries in almost every language > > you > can > > > > name, so this doesn't appear to be a big problem. > > > > FTP is another and offers the ability to transfer files without the > > base64 encoding. > > > > Ollie > > > > At 11:45 AM 6/5/2002 -0400, you wrote: > > > > >In-band transport of large payloads is something we and others have > > >been looking at pretty intensely. Obviously it would be a nice > > >thing to have, but it is also very, very difficult to do properly. > > >If you just stick base64 in an X element, you have huge problems > > >because if that takes 10 minutes to transmit, you can't send > > >anything else for those 10 minutes. > > You > > >could chunk them, but that hardly makes things simpler for the > > >client software. This also makes it massively more difficult to > > >distinguish legitimate traffic from a denial of service attack. > > >Furthermore, it means the server has to do a whole lot more XML > > >processing (which may already be a bottleneck), because all XML > > >content has to be at least checked for well-formedness. To speak > > >nothing of the bandwidth implications. > > > > > >Ultimately, I don't believe there is a satisfactory way to > > >transport large payloads in-band while keeping things simple for > > >the client. The solution to this problem will involve a more > > >complex system on the client endpoints > > >- though not necessarily in typical client software. > > > > > >-Mike > > > > > > > > > > > >|---------+----------------------------> > > >| | Andy Beetz | > > >| | <andy.beetz@clear| > > >| | swift.com> | > > >| | Sent by: | > > >| | jdev-admin@jabber| > > >| | .org | > > >| | | > > >| | | > > >| | 06/05/2002 10:29 | > > >| | AM | > > >| | Please respond to| > > >| | jdev | > > >| | | > > >|---------+----------------------------> > > > > > > > > > -------------------------------------------------------------------- > > -- > > ---- > -- > > --------------------------------------------------| > > > > > | > > > | > > > | To: "'[EMAIL PROTECTED]'" > > > <[EMAIL PROTECTED]> > > > | > > > | cc: > > > | > > > | Subject: [JDEV] File > > > transfers > > > | > > > | > > > | > > > | > > > | > > > > > > > > > -------------------------------------------------------------------- > > -- > > ---- > -- > > --------------------------------------------------| > > > > > > > > > > > > > >I've set up jabberd and got a couple of clients connecting to it > > >(winjab). I tried a file transfer which worked no problem. What I > > >saw looking at the Winjab source is that the receiver downloads the > > >file from the sender on it's own socket based connection. > > > > > >I'm just thinking that there should be a better way to do this and > > >inside the message. I'm not saying my idea is the best or anything, > > >but I do > > think > > >that it would present the client authors with less headaches. > > >Anyway, my idea is that a message element can have a child, let's > > >say attachment or even an x, which will contain the contents of the > > >file. XML can handle > > this > > >if the file is base64 encoded, as it ends up as plain text. > > > > > >Just some thoughts > > >Andy Beetz > > > > > > > > > > > -------------------------------------------------------------------- > > -- > > ---- > -- > > ----------------------------------- > > > > > > > >Clearswift monitors, controls and protects all its messaging > > >traffic in compliance with its corporate email policy using > > >Clearswift products. Find out more about Clearswift, its solutions > > >and services at www.clearswift.com. > > > > > > **************************************************************************** > > ******* > > > > > > > >This communication is confidential and may contain privileged > > >information intended solely for the named addressee(s). It may not > > >be used or disclosed except for the purpose for which it has been > > >sent. If you are not the intended recipient, you must not copy, > > >distribute or take any action in reliance on it. Unless expressly > > >stated, opinions in this message are those of the individual sender > > >and not of Clearswift. If you have received this communication in > > >error, please notify Clearswift by emailing [EMAIL PROTECTED] > > >quoting the sender and delete the message and any attached > > >documents. Clearswift accepts no liability or responsibility for > > >any onward transmission or use of emails and attachments having > > >left the Clearswift domain. > > > > > >This footnote confirms that this email message has been swept by > > >MIMEsweeper for Content Security threats, including computer > > >viruses. > > > > > >_______________________________________________ > > >jdev mailing list > > >[EMAIL PROTECTED] > > >http://mailman.jabber.org/listinfo/jdev > > > > > > > > > > > > > > > > > >_______________________________________________ > > >jdev mailing list > > >[EMAIL PROTECTED] > > >http://mailman.jabber.org/listinfo/jdev > > > > Michael Oliver > > Chief Technology Officer > > AppsAsPeers.com > > 7391 S. Bullrider Ave. > > Tucson, AZ 85747 > > 520.574.1150 > > > > _______________________________________________ > > jdev mailing list > > [EMAIL PROTECTED] > > http://mailman.jabber.org/listinfo/jdev > > > > > > > > > > > > _______________________________________________ > > jdev mailing list > > [EMAIL PROTECTED] > > http://mailman.jabber.org/listinfo/jdev > > > > > > -------------------------------------------------------------------- > > -- > > ---- > ------------------------------------- > > Clearswift monitors, controls and protects all its messaging traffic > > in compliance with its corporate email policy using Clearswift > > products. Find out more about Clearswift, its solutions and services > > at www.clearswift.com. > > > **************************************************************************** > ******* > > This communication is confidential and may contain privileged > > information intended solely for the named addressee(s). It may not > > be used or disclosed except for the purpose for which it has been > > sent. If you are not the intended recipient, you must not copy, > > distribute or take any action in reliance on it. Unless expressly > > stated, opinions in this message are those of the individual sender > > and not of Clearswift. If you have received this communication in > > error, please notify Clearswift by emailing [EMAIL PROTECTED] > > quoting the sender and delete the message and any attached > > documents. Clearswift accepts no liability or responsibility for any > > onward transmission or use > of > > emails and attachments having left the Clearswift domain. > > > > This footnote confirms that this email message has been swept by > > MIMEsweeper for Content Security threats, including computer > > viruses. > > > > _______________________________________________ > > jdev mailing list > > [EMAIL PROTECTED] > > http://mailman.jabber.org/listinfo/jdev > > > > > _______________________________________________ > jdev mailing list > [EMAIL PROTECTED] > http://mailman.jabber.org/listinfo/jdev > _______________________________________________ > jdev mailing list > [EMAIL PROTECTED] > http://mailman.jabber.org/listinfo/jdev > _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev _______________________________________________ jdev mailing list [EMAIL PROTECTED] http://mailman.jabber.org/listinfo/jdev
